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set dealing with the. Vocational Rehabilitation Program Standards 
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rehabilitation program in achieving its mandate.) Covered in the 
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system, computing .and presenting the eight performance standards, 
administering the' closure and follow-up surveys, the five 'procedural 
standards, and an introduction to the "modif ied case review schedule. 
Each section contains presentation plans and overhead 
masters . (MN) 
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PREFACE 

■ • X V ■ . 

These materials have been prepared ^or trainers who are instructing^ 
others in the use of the RSA Vocational Rehabilitation ? Program Standards 
Evaluation System and in collection of information for performance and^ 
procedural standards. These verbal and visual materials are designed to 
use in conjunction with the Trainee HandbObk . More detailed on the. stan- 
dards. system can be found in 'two earlier Berkeley Planning- Associates'!*, 
reports, t^p Program Standards Analytic Paradigm (Berkeley, Californ/3: 
June 11, 1982) and the Program Standards Guidance Materials (Berkeley, 
Calfiornia: November* 17, 1981). 
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INTRODUCTION' AND KEY TO SYMBOLS 

• — ;v — — 

* - . . • ~n * 

.the 1973 Rehabilitation- Act contained, among its'many other provisions, 
a requirement that evaluation stan4ards be devise.cK^nd implemented to 
measure" the performance of ths VR program in achieving its mandate. - Over v 
the last four years , "Berkeley Planning Associates, under contract to the 
Rehabilitation Services Administration, has developed a revised system of 
evaluation standards. Two- distinct sub-systems - of performance measures 
were developed. One, the proposed Program Evaluation Standards, evaluates 
the federal-state VR programs. The other, the proposed Project Evaluation 
Standards, measures the effectiveness of individual projects,, as well as 
aggregated program authorities funded by RSA discretionary .funds . During 
the last three, years, these systems were pretested in six model state evalu- 
ation units." .In this pretest, BPA assumed the responsibilities which will 
ultimately be under the authority, -and perhaps actual execution, of RSA, by 
providing training in the instruments and procedures for their administra- 
tipn, providing technical assistance to the states in conducting the pretest 
and analyzing their data, and providing -the basT.c reporting of the states' 
performance. x '• 

\ The final recommended Program Standards consist of eight Performance 

Standards and associated data elements; and five Procedural Standards and 

associated data elements (see Table 1). The Performance Standards pertain 

to service outputs and outcomes (e.g., coverage ,■ effectiveness , impact}, 

while the Procedural Standards pertain to service method and process (e.g., 

case handling). If implemented, the Program Standards would require a 

~~ . > ... 

revised reporting system for all state [agencies . The federal administration 

would," in turn, generate information fpr measuring the achievement of overall 
program goals and for monitoring key prpcesses which protect client interests 
In addition, the system design includes- a mechanism for analyzing and under- 
standing the factors contributing to goal achievement, and for applying that 
understanding in support of federal and state program managers and policy 
makers in decisions regarding changes in program procedures and policy, state 
needs for technical assistance, and program needs for further investigation. 
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Table 1 * ' * 

VR ProgTam Standards and Data Elements: Final Recommendations, 1981 



PERFORMANCE STANDARDS AND DATA ELEMENTS r 

1 . Coverage ' 
VR shall serve the maximum proportion of the potential ly eligible target population', subject ^ 

#to the level of federal program funding and priorities among clients. 

Clients served per 100,000 population , 
(ii) Percent severely disabled served ? 

2. Cost-Effectiveness and Benefit-Cost Return 
The VR program shall use reAurces in a cost-effective manner and show a positive return to 

■society of investment in vocational rehabilitation of disabled clients. 

(i) Expenditures per competitively employed closure * 

(ii) Expenditure per 26 -closure 

(iii) Ratio of total VR benefits to total VR costs (Benefit-cost ratio) 

(iv) Total net benefit from VR services {Discounted net present value) 

! 3. Rehabilitation Rate ^ 

VR shall maximize the number and proportion of clients accepted for services who. are 
successfully rehabilitated, subject to the meeting of other standards. 

(i) Percent 26 closures 
I (ii) Annual change in number of 26 closures 



Economic Independence 

Rehabilitated clients shall evidence economic independence. 



(i) Percent'26 closures with weekly earnings at/above federal minimum wage j 
, (ii) Comparison of earnings, of competitively employed 26 closures to earnings of employees , 

in state 

! 5. Gainful Activity - j 

I There shall be maximum placement of rehabilitated clients into competitive employment. J 

"i Noncompetitive closures shall represent an improvement in gainful activity for* the client. 

| (i) Percent 26 closures competitively employed 

i (ii) Percent competitively employed 26 closures with hourly earnings at/above federal 

1 minimum wage - * 

] * (iii) Percent noncompetiti vely employed 26 closures showing improvement injunction and 

j life status (implement after FAI/LSI pretest) 

I 

6. Client Change 

"Rehabilitated clients shall evidence vocational gains. 

! (i) Comparison of earnings before and after VR services 

I (ii) (In^addition, changes in other statuses, and functioning ability, when such measures 

| become available) 

7. Retention . 
I Rehabilitated clients shall retain the benefits of VR services. 

I* (i) Percent'26 closures retaining earnings at follow-up « 

(ii) Comparison of 26 closures with public assistance as primary source'of support at 
closure '.and at follow-up ^ 

(iii) Percent noncompetitively employed 26 closures retaining closure skills at follow-up 
"'(implement after FAI/LSI pretest) 

Satisfact ion 

Clients shall be satisfied with, the VR program, and rehibilitatcd clients shall appraise . 
VR services as useful in achieving and maintaining their vocational objectives. , 

(i) Percent closed clients satisfied with overall VR experience * 

(ii) 1 Percent closed, clients satisfied with: counselor, physical restoration, 
job training services, placement services 

(iii) Percent 26 closures* judging services received as useful in •obtaining their job/ 
homemaker situation or in current performance 
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'PROCEDURAL STANDARDS 



9. R-300 Validitv 



Information collected on clients by the R-300 and all data reporting systems used by RSA 
shall be valid, reliable, accurate, and complete. 

10. Eligibility ' 

* Eligibility decisions shall be based on accurate and sufficient diagnostic information, 
- and VR shall continually review and evaluate eligibility decisions to ensure that 
decisions are being made in accordance with laws and regulations. 



11. Timelfoc 



VR shall ensure that eligibility decisions and client movement through the VR process 
occur in a timely manner appropriate to the needs and capabilities of the clients. 

12. IWRP 

VR shall provide an Individualized Written Rehabilitation Program for each applicable 
client and VR. and the client shall be accountable to each other for complying with this 
agreement . 

13. Goal Planning . 
— i 

Counselors shall make an effort to ,set realistic goals for clients: Comprehensive con- 
sideration must be given to all factors in developing' appropriate vocational goals such 
that there is a maximum of correspondence between goals and outcomes: compfcii tive 'goals^ 
should have competitive outcomes and noncompetit ive - goals shouldjhave noncompetitive 
outcomes . * 4 
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This volume contains Presentation Plaas for training sessions on five 
topics in the Program Standards Evaluation System:" - 

AV An Overview of the Program Standards System discusses the 

standards and their data elements, summarizes the information 
requirements for performance and procedural standards , and 
illustrates the uses of the system. It is designed as a 
general introduction for all users of the system (both state 
'and RSA staff), 

B. Computing and Presenting the Eight Performance Standards is 
a step-by-step . guide for state agency and RSA staff on how 
the statistical data elements are to be computed and dis- ^ 
■* played. 

* C. Administering the Closure and Follow-up Surveys gives 

detailed information on how to conduct client • surveys to 
obtain the closure and follow-up measures called for in the 
standards. ' * 

D. The Five Procedural Standards discusses the standards for 
R-300 Validity, Eligibility, Timeliness, IWRP, and Goal 
Planning, It is designed for VR or RSA staff assigned to 
completing or using the procedural standards. , 

E, . Introduction to the Modified Case Review Schedule (MCRS) is *' 

designed for state agency staff or RSA staff responsible 
for implementing the case review. . * . 

\ ■ ' * *' 

A should be presented to 'all training audiences. It standas alone, but' 

should precede a session on £, £, £, or £. Session IE should also be 

preceded by the introduction to the procedural standards, Session £. 

Each of the Plans corresponds to %, Trainee Handbook section, but 

contains the material in briefer form for verbal presentation, (frainers 

should provide copies of the relevant sections of the Trainee Handbook 

for each person attending a training/) In addition to the verbal "script 11 

that is provided in this Plan, there are Overhead designs keyed to each 

plan. The Overhead designs are included in this volume and directly follow 

the Plan to which they correspond,. Here, the designs 'are reproduced on 
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plain white paper. The designs can be reproduced onto transparencies for 
projection,, using a photocopy process or 35 nun slides. Or, they can be 
used as the basis for preparation of flip-charts. 

The symbols in the Presentation Plans are included to alert the trainer 
to an important point, to jndicate which Overhead design corresponds to 
the topic, and to suggest special emphasis techniques. 

• Indicates a major lecture point. Be sure to include all such 
pofnts in a training session." " ' 

7 A number indicated which of the numbered overhead designs 

should be used at this point in the presentation. The numbers 
are in sequential order, starting with "1". As the "script" 
reaches a new number, the trainer should change the viewer, 
slide, or flip-chart. 

** Indicates that several of the points on one overhead should 

be introduced one at a time. The trainer may choose to cover ' 
the slide and reveal each point as it is mentioned in^the pre- 
sentation, or to point to each of these items as they are 
mentioned, for special emphasis. J 

/ Indicates^the formulas which will be presented on the over- 
head projector.. . The presentor may either read the formula 
to the audience or simply refer the audience to the overhead 
picture. 

TS These are transitional statements, leading from pne lecture 4 
point to the next. 

These plans provide highly detailed scripts to help the trainer cover 
a large amount of material in a relatively short period of time. The 
trainer should practice the presentation, using the visual technique to 
be employed in the training, and note the actual deli-very time. A column 
has been prov/ded on the left-hand side of each page to record a trainer 1 s 
own Viapsed tijne for the sessions. Overall, each session should' be com- 
pleted in approximately 45 minutes to one hour,, with the exception of the, 
overview, which will require approximately one and one-half hours. 

< »■ 
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AN OVERVIEW OF THE PROGRAM STANDARDS SYSTEM 

* • 

jl Does everyone have^a Trainee Handbook- which •■ 
looks like this? 

.#. All the information to be discussed today is 
contained in this packet 

• The entire packef may be used as a reference 

guide both during and after today's presentation, 

TS - Today's presentation has. three objectives: 



OBJECTIVES OF THE PRESENTATION - 

"To introduce you to each of .the Program Stan- 
dafds and corresponding data elements; and 

• To briefly explain how using^the system bene- . 
fits program managers and clients. 



BACKGROUND > 

'The Rehabilitation Act of 1973 included a 
provision which required that the government 
v! establish and publish a set of regulations and 
. general standards for the evaluation of the 

• 4 programs and projects effectiveness in achiev- 

ing the- Act 's . objectives / 

• Since the passage of the Act, RSA arid a number ' 
of other key actors in ttfie rehabilitation field 

• have worked to develop a 'system, which would 
respond to this mandate. Our presentation 
tod'ay begins with the process BPA undertook in 
1976 to revise the standards x 'a process which, 
has led to the current 13 Program Standards. 

TS - Let us begin now by looking' at the development 
of the Program Standards System* 
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DEVELOPMENT OF THE PROGRAM STANDARDS . 

• One of the key criticisms of the original stand- 
ards had been their apparent lack of an under- 
lying conceptual framework, ; After a careful 
review of the various alternative methods for 
developing standards, an approach which empha- 
sized program QutCfllllfi. was recommended .7 *Th is 
decision was based upon BPA's; belief that if the 
standards were to guide performance and measure 

; achievements, the standards should focus on meas- 
uring the achievement of all program' goals. , • 

0 BPA and RSA discuss^ whether various .program 
measures should be regarded as performance 
standards, procedural standards, or supportive 
* evaluation elements, p . ^ 

• The specific rolfe for each of these categories 
were: . ■>''■'*.. ? * 

Performance Standards would measure the 
achievement of desired outcomes, or the 
mission of th'ii program, 

Procedural Standards wuld address the 
^ projection of client interests by insur- 
ing key processes were ^effectively imple- 
mented, ^ 

Supportive Evaluation. Elements are aspects 
,\ of the VR process useftil in the analysis 
of performance to explain differences and 
to help to "identify program actions to . 
enhance, 

• This process and. further refinement 6f the 
standards resulted in a revised list 'of stand- 
ards, which included a total of eight performance 
standards and four procpdural standards. 

• These standards then pretested in a sample of 
six" states in order to determine if they 
adequately addressed the VR systeiA/s goats and ^ 
if^when implemented, meat the/needs of program 
managers, counselors % ana clients. 



•Time . . 

^Frojm ...... . Overhead ^ 
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Three principal purposes have underlined all 
*of the developmental work BPA has conducted 
L on the- Standards System. In conducting thp 
» t protest- and in incorporating the comments made 
r V: blithe various states and other reviewers info* 
ttofrtvised standards system, the three con- 
- > . „ cngfrs which governed our efforts were that the 
resulting standards system- be sufficient; 

** - to fjuide thjD behfiylqr^pf * state VR agencies 
. towards greater achievement^ 



'-- to make available information on the state 
• N VR agencies' achievement with respect to the 
gQa r ls and functions of the ,VR system, as 
x measured by the/ standards* data elements ; and 
v ~ y . " . ' >- 

to identify possible problems and corrective 
• actipns, whenever state, VR agencies are un- 
able to reach thetr object iveS~fo~r adhteVe-T 
merit r 1 . 



* t • These three purposes reflect thfl fact that the' 
fcj focus of the new standards system is state 
agency management improvement; and evaluation 
capacity. % ^ , • . 

** • The fedei'al ro*le- is- proposed as one of necessary 
. . data provision, the generation and making avail- 
«• able of .comparison data as appropriate, and the. 
provision of technical assistance to the state 
agency for interpreting standards data and iden- 
. tifying how to improve program performance. v 

** • The leadership role in improving state perform- 
ance is assigned to the individual state agency 
under the revised standards system. 



TS - Having outlined the fodus of the system, let 
us now turn out attention to the' specific 
standards . 
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THE REVISED PROGRAM STANDARDS *AND DATA ELEMENTS 

- BPA has recommended- a tot$l of eight Performance 

- s Standards qnd five Procedural- Standards to RSA. 

•These standards and corresponding data elements 
closely parallel the set^of standards recommended 
prior to the pretest. The niost significant 
•change was. the addition of a fifth Procedural 
Standard to capture the need for states to be 
sensitive to realistic goal-setting for clients 
and for adherence f to the policy of seeking < com- 
petitive employment outcomes when feasible. 
OtHer changes consisted of minor word changes 
within various standards and the l&flnement of 
certain data elements. 

TS - The 'following discussion briefly reviews each 
of the^l3 Program Standards . Those in the 
audience Who. wish more detailed information 
,bn the rationale behind each standard and its 
< '- corresponding data elements should refer to 

the writtjen materials' included in your present- 
ation packet, * *^ 

PERFORMANCE STANDARDS 

, ' v STANDARD 1: VOCATIONAL REHABILITATION SHALL SERVE. - 
THE POTENTIALLY ELIGIBLE TARGET POPU- 
LATION, , SUBJECT TO THE 'LEVEL OF FEDERAL 
PROGRAM/FUNDING AND PRIORITIES AMONG 
' CLIENTS. ' ■ ' , 

This standard addressed the extent to which -the VR 
/sj \ program is -serving the ^eligible target population.: 
1.. ;It is of paramount importance, to 'ensure accessibi- . 
lity of services to all. eligible, disabled. 

TS - Standard 1 has two data elements: 

Vs 1. Annual number of clients (RSAr-113) \ 

' State population (in 100,000 's) (Census data) 



/ >2. Annual number of severely 

disabled clients served •- fRSA-113) 
Annual nurtber of clients served (RSA-113) ; 



- .V 



ERIC 



Points for Presentation 



STANDARD 2: 'THE VOCATIONAL REHABILITATION PROGRAM 

SHALL USE RESOURCES IN A COST-EFFECTIVE 
MANNER AND SHOW A POSITIVE RETURN TO 
- SOCIETY OF INVESTMENT IN VOCATIONAL 
REHABILITATION OF DISABLED CLIENTS. 

This standard relates to the program's cost- 
effect^vess goal (i.e., hoW successfully did it 
achieve ~9l^Jxed objectives with the financial 
resources available) and the cost-benefit concerns 
(i.e., what was the return on the investment).. 
Specifically, it addresses the question of whether 
we are getting more put of the program than we are' 
putting into it. 
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TS - Here, we have four data elements: * 

1 . Total agency expenditures (RSA-2) 

Number of competitively (RSA-300) 
employed 26 /closures 

' - , \ ' 

2. Total agency expenditures (RSA-2) 

Number of 26 closures (RSA-300) 

3. g^ffi'? * | ■ RSA-300, RSA-2, 
(C0Sts) ] ' RSA-113, Follow- 

4*. (Benefits-Costs) * ) U P Surve y , 



STANDARD 3:^ VR SHALL MAXIMIZE THE NUMBER AND PROPOR- 
• 9 TION OF CLIENTS ACCEPTED FOR SERVICES 

WHO ARE SUCCESSFULLY REHABILITATED, SUB- 
JECT TO THE MEETING OF. OTHER STANDARDS. 

Traditionally, success in'VR has been measured by" 
the number of M 26 closures^ 11 or successful rehabili- 
tations obtained. Because a central goal of VR is - 
to rehabilitate .clients, it is essential that the 
standard? System inclyde a way of - presenting how many 
individuals were successfully, served and the extent 
to which this number increases over time. 

TS - This .standard has two data -elements : 

1., Number of 26 closures (RSA-113) 
Number of ,26/ +.28 ♦ 30 (RSA-113) 
.closures . i 

'*•» 

.2; (Number of 26 closures in current yea^r) - (rsa-113) 
(Number of 26 closures, in previous year) 1 
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10' STANDARD 4:. REHABILITATED CLIENTS SHALL EVIDENCE 

ECONOMIC INDEPENDENCE. 

This' standard 'concerns the rehabilitated client's 
ability to become economically self-sufficient. 

TS - Hqre again we have two data elements: 

Number of 26 closures with 

weekly earnings at or above 1 (RSA-300, U.S. 

federal minimum wage , Census Data) 

\ Number of 26 closures : (RSA-300) 

2. Mean weekly earnings of ; 

competitively employed 26 y ,s„ (RSA-300) 
Mean weekly earnings of. (U .S . Census Data 4 ) 
other employees in the state' 




\H STANDARD 5: THERE SHALL BE MAXIMUM PLACEMENT OF 

REHABILITATED CLIENTS INTO COMPETITIVE 
EMPLOYMENT. NON-COMPETITIVE CLOSURES 
• SHALL REPRESENT AN IMPROVEMENT IN 
. GAINFUL ACTIVITY FOR THE .CLIENT. 

This standard concerns the impact of VR jagencies 
on the clients, irrespective of whether this impact 
results in competitive employment or an improvement- 
in function and life status for'the client. s 

TS - Standard 5 has three data elements: 

/ 1 . Number of competitively erfiployed 26 T s (RSA-300) V 

• Number of all 26 closures (RSA-300) 

/ 2. Number of competitively employed 

26 closures with hourly earnings (RSA-30Q,U.S . 

at or- above federal minimum wage ; Census Data) 

Number o£ competitively employed 26 T s (RSA-300) 

/ 3. Number of non-competitive 26 f s 

witfi improvement on LSI-FAI 

measure from plan to closure, >, (RSA-300) 

Number of non-competitive 26 T s (RSA-300) 
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STANDARD 6: REHABILITATED CLIENTS SHALL EVIDENCE 
VOCATIONAL GAINS 

It is axiomatic that rehabilitated clients should 
evidence some sort of vocational gains either in 
monetary or non-monetary terms at the point VR 
services terminate. This standard assures that 
attention will be paid by the VR field to document- 
ing and seeking changes in a client's earning status, 
'functional abil-ity, or life status \ It. supplements . 
the concern for measuring post-service outcomes (as 
in Standards 3-5) by using the client's preservice , 
circumstances^as a baseline for comparison. 

TS - Standard 6 has two data elements: 

1. (The sum of closure earnings 
' for all 26 closures) k minus 

(The sum of referral earnings 

for all 26 closures) (RSA-300) 

Number of 26 closures , (RSA-300) 

2. In addition to vocational change (as measu^pd by 
the first data element), the VR program also often 
acts' as a change-agent in terms of non-vocational 
aspects of a client's life. This reality has 
resulted in the inclusion of a second data ^element 
for this standard. As with the data elements 
associated ,W£th non-competitive employment -clos- 
ures,, the methodology for assessing non-vocatipnal 
change requires further development before a 
specific computation formula can be developed, 

* ■ 



STANDARD 7: REHABILITATED- CLIENTS SHALL RETAIN THE 
BENEFITS OF VR SERVICES. 

Vocational rehabilitation programs, like all. service, 
programs, ideally strive to have the gains realized 
by their clients through program participation re- 
tailed over time. Job losses shortly following suc- 
cessful closure cah identify serious' short-comings 
in a program's service strategy and may point to an 
incongruence between program goals aria individual 

13 



Points for Presentation 

• 

client goals. Are clients being "rehabilitated" only 
on a' temporary basis, or are the gains achieved dur- 
ing the service period being retained over time? 
This question has a great degree of importance to the 
overall VR mission and thus a standard in this area 
is highly appropriate. Aside from employment measures^ 
of benefit retention, additional attention is given- 
to expanding the data elements for this standard to 
include n on -employment measures, 

TS - Standard 7 has three data elements: 

1. Number of 26 f s with earnings _^ \ 

at closure who retained or 
increased earnings at follow-up ] (RSA-300) 
4 Number of 26 closures with (Follow-up Survey)/ 
• earnings at* closure, surveyed 
at follow-up 



Percent of 26 closures with 
public assistance as- the 
primary source of support at v ' 

follow-up ; (Follow-up Survey) 

Percent of 26 closures with (RSA-300) " 

public assistance as the primary 
source of support at closure 

Number of non-competitive 26 
closures retaining FAI/LSI * . 

- closure skills . CRSA-300) 

Number of. non-competitive 26 (Follow-up Survey) 
closures surveyed at follow-up 



STANDARD 8: CLIENTS SHALL BE SATISFIED WITH THE 

VOCATIONAL REHABILITATION PROGRAM, AND 
REHABILITATED CLIENTS SHALL APPRAISE 
VOCATIONAL REHABILITATION SERVICES AS 
USEFUL IN ACHIEVING AND MAINTAINING 
THEIR VOCATIONAL OBJECTIVES. 

As an indicator of consumer appraisal of services, 
the standard on. client satisfaction with vocational 
rehabilitation services has considerable merit. 
Since client satisfaction polls usually )£fer a 
high degree of support for the program, this standard 
is viewed as being beneficial in lobbying for expanded 
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financial support at both the state and fcederal level. 
Complementing the political utility of satisfaction 
measure is the inclusion of a client utility assess- 
ment measure for this standard* The intent of this 
clause is to ensure that successfully closed clients 
'assess the utility of VR services positively in terms 
of actually having contributed to their getting a 
job and functioning in it. As a substantive rationale 
for the satisfaction standard, utility assessment 
offers a valuable erttree for probing areas needing 
program in^)rovement and for ensuring' consumer involve- 
ment in improving the responsiveness of VR services 
'to client .needs'V 

TS-- Again, there are three data elements involved:^ 

1. Number of closed clients sur- 
veyed who are Satisfied with 

. their overall VR experience (C iosure Survey) 

Number of closed clients surveyed v 



2. a Number of closed clients satis- 
fied' with their counselors 

Number of closed clients surveyed 



(Closure Survey) ' 



2.b Number of closed clients satis- 
fied with physical restoration 

services - 

Number of closed clients surveyed 



(Closure Survey) 



2.c Number of closed clients satis- 
fied with job training services • 
Number of closed clients surveyed 



(Closure Survey) 



2. d Number of closed clients satis- 

fied- with job placement services 
Number of closed clients surveyed 

3. Number of 26 closures judging 
the services they received to 
have been useful in obtaining 
their job/homemaker situation 

or in current performance 

Number of 26 closures surveyed 



(Closure Survey) 



(Closure Survey) 
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PROCEDURAL STANDARDS 
16 STANDARD 9: ft-300 VALIDITY 



. INFORMATION COLLECTED ON CLIENTS .BY THE 
R-300 AND ALL* DATA REPORTING SYSTEMS 
USED BY RSA SHALL BE VALID, . RELIABLE, 
ACCURATE, AND COMPLETE. 

This standard would ensure that state agencies main- 
tain acceptable levels of accuracy, validity, and 
reliability in the repprting .of the R-300, as well 
as other, data. By using a case review process, an 
accuracy check ^is provided between the case folder 
information, the R-300 itself, and any computer out- 
put listings .of R-300 items selected for review. I 
These items include a rangeof demographic data, 
service. data, outcome data, and cost data. 

TS -'Now, let us move along to the next Procedural 
Standard. 



17 STANDARD 10: ELIGIBILITY 



ELIGIBILITY DECISIONS SHALL BE BASED 
ON ACCURATE! AND SUFFICIENT DIAGNOSTIC 
. INFORMATION, AND VR^SHALL CONTINUALLY 
REVIEW ANI5 EVALUATE ELIGIBILITY DECI- 
SIONS TO ENSURE' THAT DECISIONS ARE 
BEING MADE IN ACCORDANCE WITH LAWS 
AND REGULATIONS, . 

This standard concerns the determination of those 
applicants who- will receive services. Such eligi- 
bility decisions must: 

• comply' with the legislative mandate, and 
4 m ensure cosit-effectiveness . 

• ■■*•> 

In short, this standard focuses on ensuring that 
clients who . are not eligible for VR services are 
not accepted, while those who are eligible are. 
provided with services. The case review system 
will be used to monitor decisions made in compli- 
ance with this standard. 



ERLC 
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18 STANDARD 11: TIMELINESS 

VR SHALL ENSURE THAT ELIGIBILITY 
DECISIONS AND CLIENT MOVEMENT 
THROUGH THE VR PROCESS OCCUR IN A 
: TIMELY MANNER APPROPRIATE TO THE 
: NEEDS AND "CAPABILITIES OF THE CLIENTS 

This standard seeks to avoid delays in the VR pro-, 
cess which may' hinder or impede the successful 
rehabilitation of the client. It requires that 
each state have a monitoring 'system to 'flag 1 those 
cases refraining in statuses. for an untimely period, 
and provides a process for evaluating each such un- 
due delay. The reasons for this system are twofold 

* 

• First, a client's attitude toward the usefulness 
of participating in VR are formed by his or her 
perception o*f the VR treatment, a perception 
often shaped by the speed with which his or her 
case is handled; and 

• Second, a correlation has been found between VR 
timeliness and clieiit outcomes. 

The Advantages of this system are that: 

. " • It allows for a flexible interpretation of 

whether a case was- handled in a timely manner 
by avoiding rigid criteria regarding the most 
appropriate. time frame for various case activ- 
ities; and 

• It ensures greater reliability than other, sub- 
jective measurement systems by requiring the 
reviewer first to determine if a delay did 
occur and then to determine the reason for a 
given delay (i.e., agency failure, client moti- 
vation, othar agency actions, etc.). 
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STANDARD 12: IWRP 

VR SHALL PROVIDE AN INDIVIDUALIZED 
WRITTEN REHABILITATION PROGRAM FOR 
' *EACH APPLICABLE CLIENT, AND VR AND 
THE CLIENT SHALL BE ACCOUNTABLE TO.. 
EACH OTHER FOR COMPLYING WITH THIS 
AGREEMENT. 

Several aspects of the Individualized Written 
Rehabilitation Program are addressed by this 
standard. These include: 

• that an IWRP be fully developed for each eligibl 
VP client; u , 

• that the plan ensure the protection of clients 1 
rights; ' 

• that the client and counselor work^together in 
developing all goals and service plans;* 

• that the client and counselor share responsibi- 
. lity for follow-through and the annual review 

of the progress and appropriateness of the 
agreement,; .and. ' 

• that the handling of any plan ( revisions are 
timely and appropriate. 

This standard will ensure compliance with the leg-* 
islative intent of the IWRP, which in turn has 
been positively associated with successful VR 
process outcomes. - 

STANDARD 13: ■ GOAL. PL ANNING 
h - — . , — " 

COUNSELORS SHALL MAKE AN EFFORT TO 

... 1 SET REALISTIC GOALS FOR CLIENTS. 

This standard concerns the setting of "realistic 11 
goals for VR clients, consistent with their capa- 
bilities and abilities, whether t$iis means setting 
competitive employment goals or sheltered or nonv / 
competitive employment goals. It addresses the 
issue of noncompetitive closure categories being 
instigated simply to. salvage "successes 11 foif # 
clients unable to meet their planned competitive 
goals. By allowing flexibility ift goal planning, 
the standard neither hinders clients from obtaining 
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competitive closures, nor classified them as 
'""unsuccessful*! should they have a competitive 
goal but a noncompetitive outcome. 

As the four data, elements for this particular 
standard indicate, the standard's objective is 
to investigate how counselors can be more effec- 
tive in the task of "fitting" clients' potentials 
to feasible outcomes. In this way, the standard 
is used appropriately to facilitate effective 
goal planning . All four data elements are included 
in the R-300 system. l - 

TS - Having reviewed the 13 Program Standards and 
their corresponding data elements, we arc now 
ready to discuss how the system operates and 
what types of benefits the system can generate 
for program managers . 

STRUCTURE OF THE PROGRAM STANDARDS SYSTEM 

• The Program Standards system has several compon- 
ents, as indicated in the illustration: 

Standards and Data Elements :< 

Process for setting performance objectives; 

Reporting System; and 

y Data-based Decision-Support System. 

As this figure indicates, all of these components 
are oriented to providing feedback to and improv- 
ing the management! of the VR program . 

• Viewed another way, these components work , 

• .together in a circular fashion to ensure that 
the information generated through the careful 
application of the standards is retained within 
the vocational rehabilitation system for the 
benefit of future clients. 

As illustrated above, the cycle proceeds as 
follows : 

Identification of specific objectives for 
the system; • . , : 

Development of reliable and valid measures 
for addressing these objectives ; 
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** Setting of performance goals and operating 1 

procedures; 

** Operation and management , of the system; 

** Implementation 'of data collection efforts; 

** * : Implementation of data reporting; 

** Performance Assessment , and. policy analysis; 

and 

** Implementation of the'-findings which emerge 

during the assessment and analysis period. 

• :; - 

TS - With this overview iii mind, we now proceed 
. to a more "detailed discussion of how the 
revised Program Standards system adheres 
to this model. H 

' , ■ ' \ 

IDENTIFYING OBJECTIVES ( 

23 • As previously discussed, the objectives of 

the standards can be summar.i zed into four . 
; concepts : 

\^ , 

** Coverage: Is :the agency adquately 'addrees- 

p ...... ing the -scope and type of needs 

, " ; . v , of its eligible target population? 

** Efficiency: Is the. agency sufficiently produc- 

. v tive given £he resources available 

• . to it?. A 

•'.'** Impact: Does the agency help to improve 

" tl the quality of life;pf the indi- 

• *, vidua! clients it sery.es? Does 

~~ ♦ the agency return more benefits 

; ' ' . v . 1 to society than the societal 

costs it incurs? 

** . Compliance: Are eligibility- decisions made 

' r \. .. in accordance with the laws and' 

' t v; regulations? ' Are all of the 

regulations being adeqvately . 
M < addressed? . , 
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..• The trade-offs between- thtese objectives are 
reflected in the range of the proposed stand- 
ards and their various data elements. In 
judging a state's performance,- it is important 
to. .bear in mind that these objectives, have no 
absolute value and that each state must weigh 
a host of political and service considerations : 
when structuring its program. Some states will' 
place greater weight on serving larger number^ 
of clients while llther states will be more con- 
cerned with the costs of their program- Still 
. other states may' be willing to incur greater 
costs providing that theij clients are indeed 
rehabilitated at the time services afe termin- \ 
ated. 

TS - ivhile these -^trade-offs may result in various 

objectives being weighted more heavily than . 
others at a given point in time, such trade- . 
offs do not alter the fact that these four 
objectives comprise the basic foundation on 

* which 4 the 13 Program Standards were built, 



^ IDENTIFYING STANDARDS (AND DATA ELEMENTS 

• The 13 Program StaTTdartfs and their correspond-, 
ing data elements have already bejen presented 

and need not be repeated at this point/ . •, 

• One important point to make at this juncture, 
however, is the fact that each of the 13 
Program Standards do relate to one or more of 
the four objectives outlined above. 

• As illustrated in this table, all of the system's 
objectives have been addressed by at least two 

of the standards. < ' ^ ' • - 

\" 

TS - Once the necessary .measures have been developed 
the next step th the process is" the determina- 
. tion of appropriate performance goals and oper- 
ating procedures • 

SETTING .PERFORMANCE GOALS AND .OPERATING PROCEDURES 

* * * »••■■' ' ' * 

• A major shift in the proposed standards system 
'is for state agencies to set their own objectives, 

in terms of levels of expected performance by. 
which the state program is to be monitored and 
held accountable. 
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This procesis is in contrast to the current system 
which uses central tendency statistics to judge 
whether a state performed adequately in the past 

• year, with adequancy being 'determined by the c / 
average performance- level for all states, . 

• The central tendency approach, while descriptive, 
does not examine the level of typical performance 
with what is reasonable of desirable but instead 

. automatically* generates failures and success among 
state programs. ■ / 

• Onee the average performance level for all states 
is determined; certain states, by definition of 
the process, are found tjo be below the average and 
certain states are found to be above the average. 

• This method does not account for the fact that 
all states may have been doing quite well, on a 

, given ^measure or that all states may have been 
doing quite poorly on anpther measure. 

• : The new system for setting performance objectives 

places responsibility within each^ state to set 
its own pbjettives for the level of performance 
to be achieved in an upcoming, fiscal year,. rather 
than continuing with the post -hoc system based 
upon national noTms. 

• A particular advantage of this' reliance on state 
agencies to set performance level objectives is 
that it permits the standards system to be used 
for monitoring and assessing the ongoing program. 
State agencies, can use their in-house data systems 
to monitor individual data items .on a monthly or 
quarterly basis, and to see if the^program is on 

^ target in terms of moving toward annual goals or 
sustaining acceptable rates of quality closures. 

• When- setting performance objectives , state agencies 
■ might: be anticipated to look at- their past perform- 
ance, at the. lpvels of performance being achieved r 
by^ other. state programs that agency staff view as 
comparable, at the performance nationwide, and at 
pending changes in state ecorfomic conditions, 
politics or client and service mix, and qther 

. unique state factors which might affect performance 

TS Once the' appropriate performance goals have been 
• set, the system- is. feady to be implemented and 
data 'collection can begin. 
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IMPLEMENT, THE SYSTEM AND BEGIN DATA COLLECTION EFFORTS 

• In order to acquire the data necessary to determine 
the extent to which each state is meeting its 
performance objectives, uniform data collection 
procedures must be developed and implemented. As 
previously discussed, one of the criteria used in 
determining the data elements for each of the 13 
Program Standards was the availability of the. „ 

data at both the state and federal levels* 

* » 

• Building upon the existing R-300 system, BPA * 
developed a number of additional data collection 
instrtgrients to complement the range of information 
currently available to RSA through the R-300. 

■ . ' 

• Any state, may use the evaluation standards system. 
The calculation of national norms will require a 
national data system. 

• Seven sources for the PERFORMANCE STANDARDS include:' 
THREE CURRENT RSA REPORTS 

• The RSA-300 Case Service Report (providing data on 
v ^individual client outcomes) ; 

• The RSA-2 Annual Report for Vocational Rehabilitation 
(providing 'data on aggregate agency expenditures); 

f and > r - \ 0. ' 

• The RSA- 113 Quarterly Cumulative Caseload/Expenditure 
Report (providing data on the agency f .s caseload flow) 

TWO BPA DEVELOPED MAIL SURVEYS ' 

• The Client Closure Survey (providing information, on 
client satisfaction with VR services); and 

• The Client Follow-up Survey (providing information 
on client retention of benefits), 

TWO EXOGENOUS DATA SOURCES 

• The annual U.S. Census- publication Statistical 

* Abstract of the U.S. (to provide data on the; current 
federal minimum wage and on state wage norms); and 
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• The . U.S. Bureau of the Census Current Population, * 
Reports/ Series P-25 (to provide state population * 

- estimates) . 

S ■ 

• RSA is currently in the process of revising the 
R-300 and : R-113 in response to OMB requirements 
In addition, the R- 2 Annual Report is being, dis-:. 
continued. Calculation of the data elements will 
depend on data avaifability at the national and 
state" ■>Jevej5 / >-\ • 

TS - Having' reviewed the data collection procedures 
.necessary for documenting all of the data 
elements required under eadh' standard, we can 
now turn our attention to the procedures thrpugh 
which RSA wili; obt ain the' data from the states . 

• One primary data source : exists for the PROCEDURAL 
STANDARDS. ' / ; ; ; _ • ' 

This data, source, the Modified Case Review Schedule 
(MCRS) builds upon the Case Review Schedule developed 
by the San Diego State University RCEP IX and is : 
a document routinely used by regional RSA officers'/ 
whenever they conduct case reviews.' 

• In order to efhsure that the Case Review process 
would be adequate to address the data needs for 
the , majority of the Procedural Standards, BPA 
developed two additional instruments for inclu- 

. si on in the MCRS. These two instruments are: 

Determining the Validity of the R-(300 Data; and 
Timeliness Assessment Instrument. 

• Because N all of these data collection efforts are 
not routinely completed and require .special .case' 
review procedures, it is recommended that RSA 

. ' conduct this data .collect^on "effort every third 
fiscal ye^r. 

• However, one of the Procedural Standards, Standard 
13, which relates *to, the correspondence between 

v; the IWRP occupational goals and final outcomes, 
uses data only from the R-300 and consequently 
can be reported annually. 
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. DAfA REPORTING' * 

• The standard/' reporting system brings together 
the various sources of standards input data so 

b that a particular- agency r s .attainment for a 
'specific time period can ]be compared to its 
objectives for the period. In addition, the 
reporting System : wall provide the program 
managers with the capability to flag and inves- 
tigate problematic attainment. To do these .two 
things, the reporting system has been designed: 

- . to keep track of past performance as well 
' as current expectations; 

k to^ present the findings in an easy to use, 
easy to understand way, without unwieldy 
* reports, emphasizing graphical presentations 
as well as plain numbers; and 

"to make sure that the reporting tff resu>ts 
occurs in a timely fashion', so "that .future 
performance &an be influenced. > v 

. • The standards system is compatible with the 
kinds of data compilations, l^jjtinely generated, 
-even now in many state agencies 1 internal - * 
informiation systems, and can be adpated by • . 
individual state agencies for their use, quite . 
independently of the development of anatiqjial 
MIS. ' v • 

• "SeVeral reports can be routinely .produced by the 

system. * ' ' \ 1 ' 

/ .■ • •• ■-. ■ " : ' , ' 4 

• First, this type of report shall show the 
achievement on each of the standards for a 

\ given agency. In addition, it will ;alsb v s'how 
thf state/ s goal for the year, its last yePar's 
performance and the previous year's national . • 
norm. 

* 

Second, reports will be developed which will 
display all agencies 1 performance on each par- * 
ticular element for both the present year as 
well as the four preceeding^ years . 

•••Finally, reports will bedeveloped which will 
summarize national performance levels for all 
agencies, and for general, combined, and blind 
agencies. 



A-20 



Time ' " 

From Overhead 

Start Projection '# ^ Points for Presentation' 



• 34 ** 



• These three types of reports will be generated 

. routinely for all of the agencies atld all "of - ' 
the data elements- In addition, RSA and the 
agencies will have the capability to use the 
MIS to generate special purpose reports and 
analyses , 

• For example, the basic reports could be run 
separately for special pqpulations and the 
results may take the form of statistical^ reports 
or graphic displays, * 

TS We now turn our attention to' the performance 
v assessment and, policy analysis phase of the 
system. 



PERFORMANCE ASSESSMENT AND POtfCY ANALYSIS 

• In many respects, this component df the system 
is the' heart of the entire process. Without 
the effective analysis of the data and the 
ability of program managers to. pin-point the 
'source of poor or outstanding performance levels 
within their agencies, the system offers VR 
little in terms of management assistance, 

• Consequently, the' standards system dffers : a mech- 
anism for investigating, the causes for problematic 

^ attainment and fbr developing corrective actions 
as part of the decision-support systefh. This 
system is described .in detail in a^special report 
developed by BPA, but can be illustrated briefly 

here, * 

.* * ' < 

• Basically, the* depisioh-support system-is designed 
to provide VR program managers with information 

j^which is : , * , ^ . ' ' , ■ ■ 

relevant to the issues (i,e,., probleriis) under 
consideration; fc 

*V quickly and easily interpretable; ' 

** timely; and . . _ s / ' : - 

** suggestive either of an .immediate policy ' 

. \ response to the proble^ or^of further inves- . 
tigaiion needed, before^ an appropriate response -' 
can be formulated. 
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• The bateic flow of the decision-support system in 
shown §n the overhead. , 

• Problematic attainment, where an agency is unable 
, to meet its agreed upon objectives <£br a particu- 
lar standard or data element, is the signal for 
the process to start. 

• Let's run through the process quickly to familiar 
ize''you with the basiq decision points: - 

• Fi^st, the progtaj) managers within RSA or 
v the states determine that a given agency * 

has not met its agreed-upon .objectives . " ^ t 

, ...... ( 

Second, the program managers investigate the 
situation to determine if, they can identify 
the. necessary corrective actions. 

'Third, if they cannot identify the problem, 
the issue undergoes moire rigorous evaluation 
research i 1 

Fourth, once the' necessary corrective action 
has .been' identified, the action is implemented . 
and the agency begins monitoring the results. 

Fifth, if the results indicate that the problem 
. 'has been eleviated,. the piroces^* stops.- However 
if the problem persists,' the cycle begins again. 

• As noted, the investigation of problematic attain- 
ment, has been broken into, two parts.; 

basic problem identification % carried, out by 
program managers within RSA and within the 
State VR agencies, using the standards report- . 
irig system plus the managers' knowledge of 
program operations; and 

evaluation research, cafried'out by % evaluation 
, . -researchers within RS£ or within; the state VR 
agencies, or by outside' consultants, using the * 
proposed MIS and other .data bases , .as Well as " 
requiring primary data* collection. ; i 
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„ t , • These two p&rts differ in who carries them out 
^ and in the time frame, in which the basic problem 

v ; identification Occurs . If evaluation research 

is required/ then .most likely corrective actions 
1 will not be possible in time for -the next cycle 

\/ of the process i \ * 

• • " ■' ■ - V 

• This lag is the reason that the investigation of 
problematic attainment is broken into two parts, 
so that timely corrective actions can be taken, 
- if at all possible. 

36 • In implementing this process, a 4 program manager 

; should keep several items iji mind: 

** 1. The first levels of the process are not 

. complex; ask questions .in the most obvdous 

..and direct manner. 



i '** 1 ^ 2. > Pursue all alternative, sources qf difficulty 

^|^ ; until you are convinced that. the solution to 
., your problem does not lie in a particular 

program area. 

** 3. Recognize the fact that corrective measures 

t may emerges, at any point in the process — 

o^ten "first level indicators, will high-light 
r - 1 v * what corrective actions need to be imple- 

mented. 

. . ** 4. Be comfortable: pushing the decision tree 

as far is possible even if the process 
results iii the development of more formal 
.-evaluation research , Useful program manage- 
v . ' -;Upnt techniques ' or jgiiide lines may emerge 

^ ^ : from simply identifying what types of new 

or expanded^ information is needed to actually 

resolve tfte difficulty. 

. .> 

TS -:This discussion has been designed tb;; provide 
you with an overview of this process . "Before 
implementing this model , you ? may want to review : 
the more detailed description included in your 
informational packet. At this time, we would 
like. to move on to the final component of the 
standards system, . -V ; 
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•3.7 • The process of problem identification consists 

of tracing the possible reason's for poor perform- . 
* " ance in a given area. ■ " 

** This process begins by organizing' a. given 

standard or data element into various com- 
* :. t . . V* ponents.' 1 

• For example, if the agency notes that it has poor 
performance oh the second^ data element for Stan- ; 
dard 2, "Expenditures per ;26 clbsures," the dif-* 

v'V-' ficulty may rest with (a) the agency having too 

few 26 closures; (b) the agency serving , clients .., 
too slowly and not terminating a Sufficient num- 
ber of clients on a regular basis;, (c) . the. agenpy. 
has recently developed a bottleneck in its intake 
process and too few clients are gaining access to 
4 the system in a timely manner; or (d) some combiri- - 

:; v -.; at ion of /the above * ■ , 

. ** Having identified several distinct alternative sources 
for a given problem; the process may ..then call ,f ox 
v t . the program manager to look to qth'ey.data elements 
\ within the system for ''clues" as-to which of the 

: V> > , ',. .. alternative* paths may ihde&d lead to the actual cause 

■ of the agency's poor performance . ,. ( ■ ; „ , 

• For example, using our case of poor performance 
on an. agency's "expenditures per 26 closure?, 

• •; y ■ an examination of data element 3(i) may indeed 
reveal that the agency is achieving too low a 
■ proportion of 26 closures . This "second level" 
indicator would be, a signal to the program manager 
that he or she will want to;<purfeue the reasons 
.•.'.»...' for the low percentage by exploring other data • 

elements or conducting more formal evaluation 
research to determine why' so few clients are 26 
closures. J- 

** Alternatively, the^ "second level" indicators might*' 5 . 
discover that certain issues arfe not the problem. 

• For example, in the above case, data element 
3(i) may indicate that the agency is indeed 
achieving a satisfactory percentage of 26 

•v . closures, a filling which might suggest that. 

the system is either serving clients too slowly 
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' or not serving enough clients. Each of these 
* possibilities can also be explored at the 

"second level. 11 ■ • 

.", • 'Third level" indicators may be needed to " - 
V . explore the selected alternative further,-; 

before deciding on a specific course of action. 
The information necessary to address these; 
issues may be found in other data elements .', 
within the standards system or may suggest the 
need for more formal evaluation research. The 
a.. detailed table in the Trainee Handbook sho^s 

" questions for one. data element, , ; * 

\ • While the implementation of this process may 

appear complex, the system's logic is straight^ 
; v . v , /. ... forward, and involves simple program comparisons 
v : which all qw VR managers to progress through^-. 

decision\tree, diagnosing problems and, using 
program information to reach conclusions about 
, a . ./ • ' the causes of various problems. ' • ' 



38 -a" : - ' - PROGRAM RESPONSES 



The key to effectively using the standards system 
as a management tool will rest with the ability 
of RSA and, the. states to incorporate new pro,ced-*< , , 
ures or. policies which may emerge from the careful 
analysis ' and review of the standards 1 data. 

The primary^ actors with responsibility for making 
changes in the standards- system are the same as 
in ..the VR system at large: Congress, OMB, Depart- 
ment of Education, RSA, Regional Offices .of Rehab- 
ilitation Services, state governments, and state 
VR agencies . ' v . < • ■ ■ - • ' 

Theses changes may involve policy 'decisions; 
federal.-. ain4 state congressional, legislation 
and regulations; resource* commitment adjustments; 
data system. revisions ; technical assistance to -\ 
the states; resektch agendas ;• university counselor 
training programs;, and coordination with other ... 
programs. ^- : . : ....« , .* m \ \' v> *' ; '' >-' ; 
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T6 r This concludes our formal description of the. 
Program' Standards- system. Before, concluding'; 1 
today's presentation, however, J would like to 
make some brief comments regarding the longer 
long-term evaluation of the standards system 
and. the reasons for which the : system may need 
S to change, over time-. ' :v 



39 . V \ , EVALUATING THE 'STANDARDS SYSTEM OVER TIME , 

• - One' of the problems with the current standard^ 

system is <t hat no evaluation of the use .pf tTie- 
standards was .included. 'In contrast ,' BPA has* - 
r *- V • - ^.established thijee* criteria for* evaluating the. 

. . v \ . . ^'''A re vis iEid"" standards "system. c . ' 

■ \ **■■>■„■"'• vThe irtpSt important evaluative criterion is 
• v * * .'„• whether the^ attainment of , the state VR agencies 
is improving, in the areas measured by the 
'■ *. t standards data elements. While it may be v6ry 

difficult to provie that the cause of the improve- 
ment was the implementation of the standards, at 
least the attainment of the agencies "after the ■ — 
implementation can be compared to their attainment 
fc * before the implementation. ■ 

** • The second evaluation criterion is whether the " k - 
state VR agencies are meeting their objectives. 
If they never meet their objectives, then the 
■ - objective * setting process is 'hot working properly . 

If they always meet t^eir objectives,, then the 
process is also not working properly . Identify- ... 
ing for which state VR agencies, for f which data ; 
-elements , or far both in combination,; '"Which objet-* 
; : ., tives are not. being mety will indicate whe^e *w ? 

; attention needs to; be' paid in the standards system, 

r : v\*--.. • v.The' third .'evaluative^ crifcefidri is whether the 
; . 4 ';' program manjagers find the system .useful . ,, Program 

managers < shdul^ be regularly" canvassed for theiac. 

• ■. recommendations. ' ' . t :* . ^ : e "\ -1 
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CHANGING THE SYSTEM . 



The key word fox the 'standards : system should be. 
FLEXIBILITY. A wide flange of factors both rt 
. within and outside the standards system may • • 
demand that changes be made in the number, the";.^ \ 
nattlre or measurement of the standards. . 

External reasons for change may include: 

the goals and >^inctions of the VR program " 
may Change, necessitating changes in t^e^ ;,>l 
standards ; . *v. ■ * . - j u 

the MIS or* other reporting within, or without , 
VR may change, changing what will be av2il-*>* 
* able for the reporting '.system; . 

^ the^ actors- and. types^ of corrective actions 
possibly* may change; ^ 

■i \ ■ 
/actions taken by -state VR agencies might push 

the .VR prdgraia in undesirable directions, as. 

state program managers try, to respond to, the & m 

standards system, tHus requiring additional 

standards or changed .expectationS'V 'and ..n^. 

the achievement of the state. VR ■ agencies may m • 
not be improving over' time.;. ' . ; 1. ■ 

Similarly, internal reasons for 'change may include 

, ■ * * •■■ 

some data elements may be^found to have lower 
data quality than is acceptable, and thus 
require new procedures' or even replacement; 

some of the data collection activities may 
require change, because of logistical problems ; 

difficulties in the reporting 'System and in 
../the reporting cucle may arise; and 

objectives being set may not "be correct!. 

As . such, RSA must monitor the operation of the 
standards System over time: In the beginning, 
the system should especially be closely monitored, 
so ft^at problems can be discovered early. And, 
RSA must be ready to change the standards system 
as; the need arises. 
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CONCLUDING COMMENTS 



• This concludes the formal presentation. As I 
/ stated, in; the beginning, all of . the .information 
; presented tbday is included in ^he Trainee 
. Handbook, 'Sedti on A. .1. ' ' 
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Vocational Rehabilitation Program 
Standards Evaluation System . 



An Overview of the 
Program Standards System 
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Trainee, Handbook- A : 



es/nta 



OBJECTIVES OF THE PRESENTATION 



Introduce the. Performance Standards and Data Elements -V 



•Explain system uses and benefits 



4u 



< 
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ROLES FOR POTENTIAL STANDARDS 



Performance Standards = Outcomes t 
Procedural Standards = Key Processes 



)rtive Evaluation Elements = Explanatory Variables 



Ai 
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1978 RECOMMENDED PROGRAM STANDARDS 



A© 



Performance Standards 

STANDARD 1: VR' shall serve the maximum proportion of the potentially eligible target 
• ^population, subject to the level. of federal program funding and 
priorities among clients, 



STANDARD 2: 



The VR program shall use ; :Fgwces In a cost-effective manner and show 
a positive return to society of investment in yocatiorwl, rehabi 1 itotion 
of disabled clients, 



STANDARD 3: VR shall maximize the number and proportion of clients accepted for 

service who are successfully rehabilitated, subject to the meeting ^ 
. * of other standards, 

r STAtJDARD h Rehabilitated clients shall evidence Increased economic Independence,. " 

There shall be maximum placement of rehabilitated clients Into com- 
petitive employment, ; Noncompetitive, closures shall be In accordance. 
. with the IWRP goal and shall represent an. Improvement in gainful 
.' activity for the client,; , ' I 

f « ,4 

6; Vocational gains shall be attributable to VR services. ,. ■ 



STANDARD 5: 



STANDAlUt Rehabilitated clients shall retain the benefits of VR services, 



STANDARD 8: Clients shall be sat Is i fed with the VR program) and rehabilitated 
,* . > clients shall appraise VR services as useful in achieving and 
maintaining their vocational objectives, 



„ 4 . 
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1978 REtOMIENDED PROGRAM ST As 



Procedural Stondards 



STANDARD 9: ' Information collected on cllei^by the R-300 r qnd al 1 data reporting • 
systems used by RSA shall be valid; Tellable, accurate, and ' 
* ; . complete, " . • / • - . ' 

!.'»• ' • ' % "s 

"' 

I ' : • ■* ' t . ' * 

' ' ■ . I ■ ■ , ' 

STANDARD 10: Eligibility decisions shall be based on accurate and sufficient 
• ' diagnostic information, (and VR shall continually review and evaluate 
eligibility decisions to ensure that decisions are being made in 
'. . : accordance with laws and regulations* 

STANDARD 11: VR shall ensure that eligibility decisions and client movement, through 
the VR process- octur- in a timely manner appropriate to the needs and- 
':. capabilities of -the clients, 

STANDARD 12 : VR shal 1 provide an Individual i zed Written Rehabilitation Program 

for each applicable client, and VR and the client shall be accountable 
to each other' for complying with this agreement, 



• a PURPOSES OF THE STANDARDS SYSTEM 

^ -• : ; — t"*7t; 



• To guide the behavior of state VR agencies toward greater 
achievement. . 



t To make available* information on the state VR agenci.es' 
achievement with respect to the goals and functions of 
the VR system> as measured by the standards data elements. 

"v" ' < 



• to identify possible problems and corrective actions/ 
whenever state VR agencies are unable to reach their 
objectives for achievement. 



FEDERAL AND STATE ROLES UNDER STANDARDS SYSTEM 

Federal Role . 
t Data feedback provision 
•* Generation of comparison data 
• Provision bf technical assistance far 
— data iranagement 
— interpretation of. standards 
improving performance 

State Role .'•.:-'•'•;;;•,•=, 'r % 

• LEADERSHIP IN SETTING PERFORMANCE GOALS 



STANDARD 1: Vocational Rehabilitation shall serve the 
maximum proportion of the potentially 
el igible target* population, subject to the 
level of federal program funding and 
priorities among clients. vX 



1 . Annual number of clients ^ (RSA-113) 
State population (in 100, 000s) (Census data) 

2. xAnnual number of severely 

disabled clients served (RSA-113) 



Annual number a/ clients served (RSA-113) 



A© 




Cost-benefit 



2: The Vocational 
use resources In a cost-effective manner 
and show a positive return to.loclety of 
Vinvestment In vocational rehabilitation 



J, Total agency expenditures 



number of competitively 
employed 26 closures - 



( RSA-2 ) 



(RSA-300) 



2, Total agency expenditures 



Number of 26 closures 



(RSA-300) 



3, (Benefits) 



('Costs ) 



RSA-300, RSA-2, RSA-1 13, 

. Follow-up Survey. 



4, (Benefits-Costs) > 
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STANDARD 3: VR shall maximize the number and proportion of 

clients accepted for services who are successfully 
rehabilitated, subject to the meeting of other 
standards, . ..»• * 




1, Number of 26 closures ' 



(RSA-113) 



Number of 26 + 28 + 30 closures ' (RSA-113) 



Rehab jlitatibn Rate 



2, Number of 26 closures in current year 
-Number of 26 closures in previous year 



(RSA-113) 



STANDARD 4: Rehabilitated clients shall evidence 
economic independence .' 



.1, Number of 26 closures with weekly v 
Economic Independence earnings v at or above federal , .:^(BSA-300-l),S, 

minimum - "'"Census Data) 



Number of 26 closures 


(RSA-300) 

■■ / 


2, Mean weekly earnings of • 




competitively employed 26s 


(RSA-300) 


Mean weekly earnings of i 


lU.S. Census Dotal 


. other employees In the state ; 





STANDARD 5: There shall be maximum placeitient of 
. rehabilitated client's into competitive 
employment, Noncompetitive closures 
shall represent an improvement in gainful 
"■■ activity for the client, 



1, Number of competitively employed 26s (RSA-300) 

.f . ■ - . ■■ 

•■ : ~~ ~s& 

Number of all 26 closf 

2. ' Number of competitively employed . 

•..26 closures with. hourly earnings . (RSA-300-U.S, 
at or above federal minimum wage Census Data) 

Number of competitively employed 26s (RSA-300) 



3, 



Number of noncompetitive 

wltjh improvement on LSI-FAj <)j e w item)" 

measure from plan to closure (RSA-300) 

Number of noncompetitive i26s (RSA-300) 



/ 




Vocational Gains 
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6: Rehabilitated clients shall evidence 
vocational gains, 



I. (The sum of closure earnings 
for all 26' closures) minus 
(the sum of referral earnings 
• for all 26 closures 



Number- of 26 closures 



(RSA-300) 



(RSA-300) 



r - •'■ .... ■ 



2, , Number of 26 closures showing 
improved functiortal ability 
and life- status 



(FAI-LSI Indicators) 



of 26 closures 



(RSA-300): 



V' ■> 
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STANDARD 7: Rehabilitated clients shall retain. the 
benefits of VR services, 



1, Number of 26s with earnings 
at closure who retained or 
Increased earnings at follow-up (RSA-300) 



Number of 26 closures with 


(Follow-up Survey) 


earnings at closure, surveyed 




at follow-up 




Percent of 26, closures with 




public assistance as the 




primary source of support 




at- follow-up 


(Follow-up Survey) 



Percent of 26 closures with ■ (RSA-300) 
public assistance as the , 
primary source of support at 
closure 



3, Number of noncompetitive 26 
closures retaining LS I I 
closure skills . (RSA-300) 

'Number of noncompetitive 26 (Follow-up Survey) 
closures surged at follow-up ' 




0 



jUffT Clients shall be satisfied with the Vocational 
Rehabilitation program; and rehabilitated clients 
shall appraise Vocational Rehabilitation services 
as useful in achieving and maintaining their ■ 
vocational objectives, 



1, Number of closed clients surveyed 
' who are satisfied with their overall 
' VR experience 



/ 



a,. . c; a i; e f^; nn , — — — to (ClOSUTe SUW) 

V/iieni oaii&rauiun , WlimKar nf r inc fl H niontc cnn/pwwl 



number of closed clients surveyed 

Nqmber of closed, clients satisifed 
with their counselors . 



ier or closed clients surveyed 



(Closure Survey) 



(Closure Survey) 



Kumber of closed clients satisfied 
with phys ical restoration services 

number of closed clients surveyed 

Number of closed clients satisifed 

with Job training services • V 
. . (Closure Survey) 

Number of closed clients surveyed 



STANDARD. 8 (Continued) 



Number of closed clients satisfied 
with Job placement services 

■ — ; — (Closure Survey) 

Number of closed clients surveyed 

Number of 26 closures judging the 
services they received to have been 
useful In obtaining- their job/ 
homemaker situation or iji current 
performance 



R-300 VALIDITY 



Information collected on clients by the R-300 
and all data reporting systems used .by RSA shall 
be valid, reliable/) accurate, and complete, 



ELIGIBILITY 



STANDARD 10: Eligibility decisions shall be based on accurate 
" and sufficient diagnostic information, and VR 

shall -continually review and evaluate eligibility 
. decisions to ensure that decisions are being made 
in accordance with laws and regulations. 



VR shal l ' ensure that etlgibYl Vty decisions 
anjd client movement' through the VR process 
occur In a timely manner appropriate to the 
needs and capabilities of the clients. 

. ■ *.V:.'V- .* 
'""•<*,•••'•••'••• 



IWRP 



STANDARD 12: VR shall provide an Individualized written 
rehabilitation program for each applicable 
client, and VR and the client shall be 
accountable to each other for complying with 
this agreement. 



STANDARD 13: Counselors shall make an effort to set, ; 
realistic goals for clients. 1 Comprehensive 
. consideration must be given to all factors 
In developing appropriate vocational goals 
• such that there Is maximum correspondence 
between goals and outcomes: 1 competitive ^ 
goals should have competitive outcomes anf 
noncompetitive goals should have non- ' * 
competitive outcomes, ■ 

I of 26 closures wi th cd^etltlve^QQl AND competitive outcomes 

# of 26 closures „, X*fe v 

f of 26 c|sures with 'competitive goals BUT noncompetitive outcomes 
f of 26 closures 

t of 26 closures with noncompetitive goals AND noncompetitive 

outcomes, ' X • > 

— — — — \ 1 . i 

I 'of 26 closures 

t of 26 closures with noncompete goals BUT competitive outcomes 

— — — — 

# of 26 closures 



OPERATING MODEL FOR THE REVISED PROGRAM STANDARDS SYSTEM 



Identify 




Policy Analysis 



Coverage: 



Efficiency: 



Is the agency adequately addressing the 
scope and type of needs of its eligible 
target populations? 

* Is the agency sufficiently productive, 
given the resources available to it? 



Impact; 



Does, the agency help to improve the quality « 
of life of the. individual clients it serves? 
Does the agency return more benefits to 
society (in terms of wages, taxes and other 
benefits) than the societal costs it incurs 
(e.g., tax revenues expended)? 



Compliance; 



Are eligibility decisions made in accordance 
with the -laws and regulations? Are all of the 
regulations being adequately addressed? 



..Two* 



PROGRAM STANDARDS AND THEIR RELATIONSHIP TO SYSTEM OBJECTIVES 



Standard 


Coverage 


Efficiency 


Impact 


Compliance 


STANDARD 1: Coverage , v> . 


/ 









STANDARD 2: Cost-Effectiveness and 
Benefit Cost Return 




/ 

V 






STANDARD 3: Rehabilitation Rate 






1/ 




STANDARD .4: Economic Independence 










STANDARD 5: Gainful Activity 










STANDARD 6: Client Change 






V 




STANDARD 7: Retention y 








• • 

fi' 


STANDARD 8: Satisfaction 


V 




V 
. / 


; ; 

a 

i 


STANDARD 9: R-300 Validity 










^STANDARd 10: Eligibility 


■ / 






✓ 


STANDARD 11 : -limeliness 

r >$. . • • ■ 

W 4-" 






/ 


— - — y 


■uu' ■ — — ^— 

•STANDARD 12: IWRP 

iff 






/ 


i/ 


STANDARD 13: Goal Planning - * 
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Proposed 

STATE DETERMINED 4 
Performance 
GOALS / ' 




TWO ITOODS 



RUINING PERFORMANCE (foALS 



A© 



Current 



Central Tenqency 
Determined 
GOALS 



a - 



Pros:-* Allows for careful examination 
of what is an appropriate or 
desired outcome 

->flaces responsibility for setting 
performance goals at the individual' 
state level 

-> Allows. for all states, to be 
jm successful 

^Allows for- standards to be used as 
n an effective management tool ^ 



Cons: 



-> Descriptive 
-> Simple 



-> No exaif fiatilpn of appropriate or 
desired performance levels 

-» Generates automatic . successes 
failures '-v 

-> Places responsibility for setting 
, .performance goals at the federal 
.level 



a J 
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FACTORS STATF MAY CONSIDER IN SETTING PERFORMANCE GOALS 



• Past performance • 

• Levels of performance of other, comparable states 
v Nationwide performance 

•' Pending changes in the state's economic conditions, 
politics, client and service mi*, other relevant 
variables . 

• Unique state conditions 



DATA SOURCES FOR PERFORMANCE STANDARDS 



• R-500 Case Service Report . 

Used for" Standards 2, 5, 6, 7 

• R-2 Annua 1 - Reports 

" V Used for Standard 2 

• R-113 Quarterly Report ' 

Used for Standards 1 , 2, 3 

• Closure Survey . 

Used for Standard $ 

• Follow-up Survey 

iy r ^Used for Standards 2, 7 

• Sttftisjical Abstract of U.S. 

Used for Standards kj 5 

• Current Population Repprt 

Used for Standard 1 



; ^ - MTA SOURCE: FOR PROCEDURAL STANDARDS 



A@ 



• Modified Cose Review Schedule (MCRS) 
. • • Used for Standards 10, 12 

Instrument* to Determine the Validity of the R-300 Data 
* *< " < Used, for Standard 9 '. * 



• Jimeliness Assessment instrument 

Used for Standard 1 1 

r 

• R-300 Case Service Report 

; ^ v fcteed for Standards 9/ 13 
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DATA REPORTING SYSTEM OBJECTIVES 



fcrkeep t.igck of past perfonnance as well as 
current expectation^; n - ' 




• To* pre^nt the : f indlngs In an easy to use, easy" 
■fi to undersfdnd way., without unwieldly reports,, 
emphasizing #a^^:al presentations as well as 
plain numbers; a - [ 



To make stfre that the reporting of results occu% 



in a timely "fashion/ so that future performance % 
can be ipfluenced.. , ... , • 



mm : > 



. 'achievement on performance standards 



All 



■r 



r 



'■ft 



>. ^ ^ fVeor:-- 1981 . 
^ • • StSte: California-. 



■ t 



1, COVERAGE 

in 




1 ,?, 4.v 



85 



if. 




0 
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Yectr 



XX, XX' 



Jills 

Year's 

Goal 



•f % 



1980 * 



I98d y Norm 



XXX.X XXX, X XXX, X XXX, X 



XX, XX XX, XX : XX.XX 



my 



0 - 



...v . \ 




SI 



So 



1 



ACHIEVEMENT ON PERFORKflHCF STANDARDS: STATE COMPARISON . / . 

■ • fiFNFRAL AND COMBINED AGENCIES 

r • * 

Standard: 1, Coverage , - . 

Data Elements: (i) Clients Served per 100,000#opulatlon 

National Norm: xxx.x v 



Ik 



This • r; , 

Year's ( - ; . <? t 

mr Goal v 1980 191. \M* v 'M 



xxx.x XXX.X ' XXX.X 3#S XXX, X ^ XXX.X 



xxx.x xxx.x XXX.X XXX.X .XXX.X XXX.X" 



xxx.x px.x xxx.x sxxx.x ta.x xxx;h ■? 
* .:: ^ 'v. xxx.x xxx.x / XXX.X XXX .x^ xxx.x- xxx.x • 



* ' f Mx^'xxx.x* ..xxuBI^ 'XXX.X .XXX.X 





■V*' 



- ATH I FVFMFNT ON PERFQRMAMf F STANDARDS •, 
ALL AGENCIES 



Year:, 1981 



Ml 
Norm 




tverage, - X ?' ■ ' 

(1) Clients served per-100^000 
• J ^population ;'■ •. . ; 

(1 lucent severely disabled " n 
^served' * , , " v 



•3 




Gen- 
eral 



Blind 



Com- 
bined 



xxx, x . xxx.x xxx.x- xxx.x ■ 



xx.xx '. xx, xx ixux XX, XX 



■3 » 



•S3 



I 



It.- 



0 
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DECISION SUPPORT SYSTEM 
' PURPOSE OF DATA 



o provide VR program managers with information which is 

» ■ , ■ ■ 

• RELEVANT ' - 



QUICKLY INTERPRETABLE 



TIMELY * 



SJ$6ESp^ OF ACTION 




j3 



the Plow of the decision-support system 




^ First-level questions are not complex 
•v Pursue all alternatives to fullest extent 



» Correct ^;e measures mav emerge at any. stage 
In the process/Implement them when you find 
^ them 



if tb not be afraid to suggest more complex 
jjT evaluation research,, but also use the - 
rprocess t;o Identify new ways to resolve 
poor performance - ; 



SOS. 



rji ,r K 



v Problem : 



Action; 



Expenditures/ 
Rehabilitation 
too high 




Too low o success rate? > Improve rehabili 



. «■ 



Too few c 




served?" 



tatlon rate j 

v. 




■* Client process^ too slow?— — > Identify slow* 

and expedite- 



* Increase Intake 



CRITERIA FOR EVALUATING THE PROGRAM STANDARD? 



Performance of state, agencies is improving as 
measured by the Standards ... 



State agencies are better at rfieeting their stated 
objectives '• ' % 



Program managers, f ind the system useful 



■fa .1 '■ \ ' • 

' ■ ' CHANGING THE STANDARDS SYSTEM ' , • 

« , ■ ■ 

External Change Factors 

• The goals and functions of the VR program may changed . necess i tat i/ig , changes 1 
In-the-standards . . , 

• The HIS or other reporting within or without VR may change, changing what will 
be available for the reporting system ' , 

• The actors and types .of corrective actions^possibly may change 

• Actions taken by state VR' agencies might push the VR program in. undesirable 
directions, as state program managers try to respond to the standards system, 
thus requiring additional standards or changed' expectations ' r ' 

. * . ' " * *■ 

• The achievement of the state VR agencies may ^ot' be improving over time , ' 



Cl» THE STANDARDS SYSTEM (continued) 



Internal Change Factors 

' t 1 4 

• Some data elements may be found to have lower data .quality than Is acceptable, 
and thus require * new - procedures or even replacement • ' . ' 

. ... N s , . 

• Some o'f the datp collection Activities may require change, because of logistical . 
problems , . .; ; : - / . v 

• Difficulties in the -reporting -system and In the, reporting cycle 'may arise' . 

* j , ' : • .',<'.. 

• Objectives being set may not be correct . " 
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1 VR PERFORMANCE STANDARDS: THEIR RATIONALE AND DATA* 
. , ELEMENTS • ' ■ 

. r" % Does everyone have a packet which loolcs like 
this? * 

• ■ All the information to be discussed today is 

contained, in this packet. Use it as a refer- 
. a ence guide both during ajid after the present- 
s • »' ation. 

TS. - Today's presentation' has two objectives: 

2 % _ OBJECTIVES OF THE PRESENTATION ' 

First, to, familiarize VR staff With the rationale 
< for/each of the eight Performance Standards ; 

• - Second, to outline the rationale .and data source 

for all; of the data items proposed for ,each stand- 
. arri ; and , , > 

• Finally, to present, for a s.an\pl'e standard, how 
onte would compute the data .elements . 

~ ■ v 

3 OVERVIEW OF PERFORMANCE .STANDARDS /' 

• VR Performance Standards are concerned with: 

** 1) Coverage: Is the agency adequately address- 
ing the^scope and needs of its 
~ i - „' eligible target populations? 

** 2) Efficiency: Is the agency sufficiently pro- 
ductive, given 'the resources 
available to it? ■ 

3) Impact: Does the. agency improve the quality 

of life of the individual clients 
it serves? Does the agency return 
more- benefits to society. than the 
societal costs it incurs? » • 

* Each of the eight performance_standards address one 

of these concerns. In the fol^wing presentation, 
we will first explore the rationale for each of the 
performance standards and their corresponding data 
elements. For each ^iata element, the source of the 
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Points for Presentation 



data will also be indicated on the overhead displays. 
We will <then discuss how one would actually compute 
the suggested data elements for a sample standard. 

TS - Now, let us look briefly at the rationale and 
data eTements for each of the eight perform- 
ance standards. 



-STANDARD 1: VOCATIONAL REHABILITATION SHALL SERVE 
THE POTENTIALLY ELIGIBLE TARGET POPU- 
LATION, SUBJECT TO THE LEVEL OF FEDERAL 
PROGRAM FUNDING^ AND PRIORITIES AMONG 
* CLIENTS. 

This standard addresses the extent to which the VR 
program is serving the eligible target population. 
It is of paramount importance to ensure accessibility 
of services to all eligible disabled. . 



TS - Standard 1 has two data elements: 

• The first data element addresses the numb.er of 
..clients annually-, served per 100,000 state popu- 
lation. Although this data element does not 

■ provide a true estimate of the level of coverage 
of the eligible target population, it provides 
an adequate proxy measure of the target population 
in terms of the total state population. 

The formula for this element is: 

Annual number of clients ... i (RSA-115) 
State population (in 100,000 's) (Census data) 

• The second data element concerns the percentage 
of severely disabled clients served 'in 'a 'given . 
year. Because the proportion of severely dis-^ 
abled Within a caseload cap reasonably be expec- 
ted to impact negatively upon a state, agency's 
caseload size and on its total costs, the pro- 
portion of th'e caseload that is severely disabled, 
must be taken into account to effectively assess 
coverage ■ , ■ 

The formula for this dnta element is: 

Annual number of severely 

• disabled clients- served (RSA-113) 

Annual number of clients served (RSA-113) 
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STANDARD 2: THE VOCATIONAL REHABILITATION PROGRAM 

SHALL USE RESOURCES IN, A COST-EFFECTIVE 
MANNER. AND SHOW A POSITIVE RETURN TO 
SOCIETY OF INVESTMENT IN VOCATIONAL 
. REHABILIATION OF DISABLED CLIENTS . 

This standard relates £0 the program's cost- 
effectiveness goal (i.e., how successfully did it 
achieve de-Sired objectives .with the financial 
resources available) and the cost-benefit concerns 
(i.e. , what was the return on the investment). 
Specifically, it addresses the question of whether 
we are getting more' out of the program than *we are 
putting into it. 

TS -He.re, we have four data elements : 



• The first element compares total agency expendi- 
tures to the number of competitively employed 26 
closures. It applies the most stringent criteria 
to the measurement of- cost-effectiveness by focus- 
ing on only those 26 closures who are competitively 
employed at the time services terminate. White 
this data element closely. parallels element 2(ii), 
expenditure per 26 closure, it is included because 
of the long-standing consensus that .competitive 
employment is the highest quality and most desir- 
able type of closure obtainable. ' 

The formula for this data element is: 

* Total agency expenditures' (RSA-2)* 
Number of competitively ■ t (RSA-300) 
employed 26 closures , . : ■! t 

• The- second cojst-effectiveness measure relaxes the 
measurement criteria, assessing Rvalue to all types ^ 
of rehabilitations. It recognizes that some 
clients are not capable: of achieving competitive 
employment and that other employment outcomes can 
represent achievement commensurate with these 

^Jfelients 1 abilities. This data element compares 
total, agency expenditures to all 26 closures, thus 
Capturing the effect of gainful activity, whether 
it) lies in the realm of competitive or noncompeti- 
tive employment. 

The formula for this data element is: ^ 

Total agency expenditures- (RSA"-2) 

y Number of 26 closures (RSA-300) 

IUj > . > 



The third and fourth data elements are very 
' similar in cpncept and, therefore , will be 
discussed together. Unlike cost-effectiveness 
measures, which determine the unit costs for 
achieving a given objective (such as- costs per 
competitive closure), benefit-cost models esti- 
mate total benefits and total costs in terms of 
dollars. These models are neutral with regard ' 
to type of delivery strategy. As such, they do 
not penalize agencies *wbich choose, to spend more 
per client in order to produce better results. 
Because of their surface simplicity, and because 
they are a popular sophisticated Analytic tool 
for evaluating program worth, benefit -cost mea- 
sures of the VR system are included in the Per- 
formance Standards. 

Both of the benefit-cost data elements proposed 
for this standard use the discounted present 
value of social benefits and costs, and rely 
upon the same components to arrive at benefits' 
and costs. These components are as follows: 

Benefits 

-- discounted value of paid earnings; 

-- change in output of homemaker closures;. 

— chrnge in otltput of unpaid family wbrkers ; 

-- change in "after houns woTk n (e.g, , home- 
making tasks performed by wage-earning 
rehabiiitants); 

-- fringe benefits; - 

change in output of families of rehabiiitants 
(as a result of rehabiiitants assuming home- 
maker tasks) ; 

-- reductions in public assistance benefits; and 

V * 
repeater costs' (a "negative benefit 11 ). 1 

Costs » 

total program co^ts during the fiscal year, 
minus carry-over costs and maintenance costs; 

-- costs borne by parties other than VR; 

research-, training, and demonstration costs; 

--'benefits foregone by clients during partici- 
pation in VR services (i.e., any wages and 
fringe benefits foregone by clients with 
earnings at referral); and 

-- cliertt-borne costs for VR services. 



' Points, for Presentation , - 

The fomiula and source for these two data elements 
are as follows: ; 

2-fiiiV benefits] . 
l - } C Costs ) \RSA-300, RSA-2, RSA-113, 

* j Fol low-Up Survey 

2 Civ) •(Benefits-Costs)J 1 



VR SHALL MAXIMIZE THE NUMBER AND PRO- 
PORTION OF CLIENTS ACCEPTED FOR SERVICES 
WHO ARE SUCCESSFULLY REHABILITATED, ■ 
SUBJECT TO THE MEETING OF OTHER STANDARDS. 

Traditionally, success in VR has been measured by the 
number of "26 closures," or successful rehabilitations 
obtained. Because a central goal, of VR is to. rehabil- 
itate clients, it is essential that the - standards 
system include a way of ^presenting how many individuals 
were successfuly served and the extent to which this 
number increases over -time. 

*> ■ 

TS - This standard has two data elements : 

• The first data element provides a (straightforward 
measure of an agency's success in^rehabilitating 
the clients it apcepts for services. The data 
element focuses on the proportion of clients,, 
accepted for service . (i .e . , excluding 08' s) who 
are successfully' rehabilitated. . 

The formula for this dat^ felement is: 

Number of 26 closures (RSA-113) 

Number of 26 + 28 + 30 closures (RSA-113) 



• The second data element attempts to assess an 
agency's success in maximizing the number of 
•clients, ^accepted for services, who- are success- 
fully .rehabilitated. y The measure uses the state 
agency's prior performance as .a baseline for 
-'determining success in "maximization." 'An agency 
is judged\to have maximized .the number of rehab-i • 
ilitarfts if *ithas . increased "the number of 26 / 
closures by,- some previously specified amount, "as . 
set by the state agency, in* conjunction with RSA, 
" ' / 

The formula for this data element, is: ' ' ' 

(Number of 26 closures in current 
year) " - (Number of '26 closures 
.in previous year) 



STANDARD 3: 

4 



(RSA-113)/'. .'. 
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STANDARD 4: REHABILITATED CLIENTS SHALL EVIDENCE" 
ECONOMIC' INDEPENDENCE • " - 

This standard concerns the. rehabilitated client\s 
ability to become Economically self-sufficient. 

TS - Here again we Wave two data elements: . 

• In addressing economic independence, the logical 
, ' place to look is to wages. The first data ele- 
*" ment assesses wages as they compare to the federal 
minimum wage. The normative implications of this 
data element are that .a disabled* person should be^ 
expected, under equivalent circumstances, to make 
at least the minimum wage required by law, ThiW 
data element uses the weekly minimum wage figure 
as "the standard rather than the hourly wage, 
because the former mpre accurately captures the 
concept of this standard.' Whereas ^hourly wage 
iiklicates a measure of the employee 1 s worth to 

employer'/ total earnings is a better indicator 
"""or the employee 's- finaTjsiaT well-being. If an 
employee is able to work V*£ly five hours a week, 
his/her economic ^condition will be affected fyy 
this as well as by the. hourly rate. 

. ■ . # 

The formula for this data' element is: 

Number of 2f6~closures with weekly 

earnings at or above federal min- (RSA-300-U;S . 
imum wage , " . [_ Census Data) 

Number -of 26 closures* ' ! ~ m : (RSA-300), 

• Tfie second data element controls for state-to-state t 
variation i'n earnings levels, whereas ^the first 
data element does not. In some respects, this is 
a more comprehensive indicator than the first data 

♦ . element because it provides an. estimate of a clients 
"standard of 'living' 1 relative to other persons in * 
his or "her state. 

The formula for "this data element is:. w 

Mean weekly earnings of • " 

competitively employed 26' s (RSA-300) 

Mean weekly earnings of (U.Sv Census Data)' 
other employees in the state " <- * 
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18 STANDARDISE 'THERE SHALL BE MAXIMUM- PLACEMENT OF' 

V ■:; REHABILITATED CLIENTS'.' INTO COMPETITIVE 

; • ' ' .w . ■ , / EMPLOYMENT. . NON-COMPETITIVE 'CLOSURES 

* . "V, SHALL REPRESENT AN IMPROVEMENT IN £AIN- ' 

... ^ \\ < • _ ."V FUL ACTIVITY FOR THE CLIENT. > ; 

. . ■ "Y "TCis Standard conofejjs the impact of VR agencies on 

„ the clients,. irrespective of whether this impact 

'9. . results in* competitive employment or an improvement 

1 r . t *iiU function and life- .status fox the clients. * . ; 

. . : -' v.'-';. , . 

:-.' v ' . \ * ',. ' • TS Standard 5 Jias three* data elements: 

• ^ . / ' , ' ■ ' ' x • / J '\ 

19 : . . ^ • This standard; s bias toward competitive £nfploy- 

- Uifejit reflect^ the belief, that vocational' rehab-, > 

; ? ilitjatipn should foqus on employment , preferably 

,1. competitive employment . For a standard emphasiz- 

ing maximum placement 'into "competitive employment,, 
$ \ "perhaps the most obvious data element is to deter- 1 

^mine tne*proportion. pf; 26 closures .placed into 
* .•■ ' competitive employment*. _ . 

t "The formula for the data element is:' . - 

• " i) . * / .. Number of competitively etiiployed 26 's ^ (RSA-300) 
^ ' ^Nujjjber all 26 closures '(RSA-300) 

20 : . • The second data J element .applies more stringent. ' v 

criteria to the measur,emeht -of' "maximum placem&nt - 
. * ■ °f rehabilitated clients into competitive employ-- 

. ,ment It compares the number of competitively 

, ' employed 26 closures with hourly earnings at or ;, 

. , , above the federal minimum wage to the total number 
, ; of competitively* employed 26 closures. As in the 

" '* first data element for Standard 4, this data ele- 

ment implies that a disabled person in the compet- 
m itive labor market should be expected to earn at 

least the federal minimum wage. Unlike the prior 
' „ data . element , however., this, measure represents an 

-employee's worth to the employer, as determined 
* " by the client's hourly wage.v Thus, this data ele- 
. * ment provides a measure of the value of rehabili- 

tated VR clients who are in the competitive- labor - 
. market relative to the federal minimum Wage. 

^ The formula for this data element is: 

. / . r Number qf competitively employed,, 4 * . 1 ■ 

' a . ''.26 closures with hourly earnings (RwSA-300-U.S . 

. . at or above federal minimum wage ' ■ Census Data) 

' ; ; % - Number of competitively employed 26 f s CRSA-300) 
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^ „•> 21- • Closures into non-competitive employment may be 

.0?. * legitimate for certain clients, but in orde,r to' 

-•■ * . --. " - attribute any credit to VR for "rehabilitating^ 

. clients into non-competitive employment, th$re 

t must be some indication that VR helped improve- 
those tlients 1 capacities for 'gainful activity.- 
The third' data element will use infdsanatiqn 
• - - gathered on events at acceptance and^at 7 closure, 

" % - - ■ , using elements o£ "the Functional Assessment./ 

• Inventory ^(FALX-and' Life Status Indicatory (LSI) 
instruments which wi;ll be added to the client's 
RSA-300. RSA is currently undertaking a pretest, 
of the FAI and LSI items -to determine. which spec- 
' ific items* to. include on the RSA-300. ! 

' The formula for this data element is: 4 

• * ■ v Number of non-competitive 26's' 

with improvement on LSI-FAI 
measure from plan to closure • (RSA-300) 
Number of non-competitive 26 , s (RSA-300) i 

■22 STANDARD 6: ' REHABILITATED CLIENTS SHALL EVIDENCE 

.. .VOCATIONAL GAINS' '' ' * 

It is axiomatic that rehabilitated clients Should 
evidence some sort of. vocational gains either in * 
monetary or non-monetary terms at the point VR 
services terminate. This standard .assures that 
•attention will be paid by the. VR* field *to document- 
, ing and seeking changes in a client's earning status, 

^ • functional ability, or life status.. It supplements 

1 • * the concern for measuring post-service outcomes (as v 

dri Standards 3-5.) by using the client's flreservice 
circumstances as a baseline for cpmparispn. 

, _ TS' - Standard 6 has two data elements 0 

23, - • The fi^st^ data element is included -because wages 

* ate the nibst .straight-forward indicator of 
^ vocational change. Weekly earnings ares used tp 
••••:> - ' measure the change in Si client's wages 'which 
_ r occurred during the periqd of time he or she 
- . ■ ■'. received, VR' services . ' 

* The formula for this data element is: ^ 

,„ ~y. (The sum of closure earnings 

for all 26 closures) -minus 
* * (the sum' of Referral* earnings' ' 
r • for all 26 closures^ . . , - (RSA-300) 

Number df 26 closures : ~ ~ ! (RSA-300) * 
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• In addition* to vocational change (as measured by^\ 
the first data element) the VR program also often ) 
acts as a change-agent in terms of non-vocational 
: 0 aspects of a client's life. This reality has 

resulted in the^ inclusion of a second data element 
for this standard. As with the data elements 
associated with non-competitive employment closures, 
the methodology for assessing non-vocational change 
requires further- development before a specific 
computation formula can be developed. 

STANDARD 7: REHABILITATED CLIENTS SHALL RETAIN THE ' 
BENEFITS OF VR SERVICES 

Vocational rehabilitation programs, like all service 
programs, ideally strive to have the gains realized 
by tjieir clients through program participation re- 
tained over time. Job. losses, shortly following- 
successful closure can identify serious short-comings 
in a program's service strategy arid may point to an 
incongruence between program goals and individual 
client goals. Are clients being* "rehabilitated" only 
on a temporary basis or are the gains achieved during - 
the service period being retained over time? This 
question has a .great degree of importance to the overal 
VR mission and thus a standard in this area is highly 
appropriate Aside from employment measures of bene- 
fit retention, additional attention is given to 
expanding the data, elements for this standard to 
( include non- employment measures. 



TS - Standard 7 has three data elements: 

• As noted, retention of benefits gained through VR 
services is -very important both to the individual, 
client and to the overall effectiveness of the , ... 
program. The first data element looks it reten- 
tion .of wagqs earned as* one of the most important 
benefits obtained from'VR. >' ' 

The formula for this data, element is : . 

Number of 26 f s with -earnings 

at closure who retained or , 
. increased earnings" -at follow-up * (RSA-3Q0) :<> 
Number -of '26' closures with* * .^(Follow-Up Survey) 
earnings at closure, surveyed 
at follow-up ' h . m \ 
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: 27 ■ -\ The seconcTdata element -provides a deeded di.men- 

, * sion in assessing benefit-retention for non- 

competitive^ as well' as competitively placed 
. successful closures. Here benefits are proxied 
■ „ r by measuring the. extent of the clients' 'use of 

' ' ' " * . ' • public resources. By focusing on the degree to . 

v < ■ ' < • 'Which there is a.reduced need for public assis- 

, tance> an emphasis is given to the economic self- 
sufficiency of the client. an- terms of stability 
. ■ ' H r or improvement; ' ^ 

$ This data element requires a new definition of 

• • Jfc '/primary source of support' 1 were "source of sup- 

™ port" is broken into only two categories (public 
" k versus private) and* where primary is taken to-' 
. •• • , mean the ;SouVc6 supplies 51% or more of 3. per- 
* .. son's total monthly support. 

'The formula for this data element is:' *** 

i. /. Percent of 26 closures with * • 

- .- public assistance as the 

Jttfe' primary^ -source of support > . 

at follow-up • v (Follow-up Survey) - 

Percent of 26 closures with . \ (kSA-300) ~~ - 
public, assistance as the 
primary source of support at ^ 
■ 4 ■ ' ■ closure 

28 • Retention of functional and life status . benefits 

is equally important as the retention of voca- 
tional benefits, particularly in the case of non- 
competitiyely employed 26 closures for whoip non- 
vocatiohal improvement may be a primary benefit 
* - derived. f torn participation in VR services. The 

third data element updates the information pro- 
vided by data element under Standard 5, and will . 
use the same FAI and LSI data items used for the 
Standard 5 data elements.. However, for the pur- 
- '■ poses of this data element, the > FAI and LSI items 

will ne§d to be modified into a form suitable for 
self-administration by the clients, via the Follow- 
up Survey. The specifip items and their forms .will 
be_ determined after completion of the RSA.'s FAI/LSI 
pretest.. Once implemented, the data elejnent will 
have the following formula and data sources: 

/ Number of non-competitive 26 
closures retaining LST/FAI 

d closure skills (RSft-300) - 

\ ' Number of non-competitive 26 (Follow-up Survey) 

closures surveyed at follow-up 
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. STANDARD 8: CLIENTS SHACL BE SATISFIED WITH THE 

VOCATIONAL REHABILITATION PROGRAM, AND 
REHABILITATED CLIENTS SHALL APPRAISE 
VOCATIONAL REHABILITATION SERVICES AS 
>• USEFUL IN ACHIEVING AND MAINTAINING 
THEIR VOCATIONAL OBJECTIVES 



As an indicator of consumer appraisal of services, 
the standard on client satisfaction with vocational 
rehabilitation services has considerable merit.' 
Since client satisfaction polls usually "offer a 
high degree of support for the program, this stan- 
dard is viewed as being beneficial in lobbying for 
expanded financial support at both the state and 
,federa,l level. Complementing the political utility 
of satisfaction measure is the inclusion of a client 
utility assessment measure for this standard. > the 
intent of this .clause is to ensure that successfully 
closed, clients assess ti\e utility of VR services 
positively in terms of actually having contributed to 
their getting a job and functioning in it. As a 
substantive rationale for the satisfaction standard, 
iitility assessment offers a valuable entree for prob- 
ing areas needing program improvement and for ensuring 
consumer involvement in improving the responsiveness 
of VR services to client needs. 

TS - Again, there are three data elements involved: - 

• As one of the data elements of the original nine 
standards, retaining overall satisfaction as a 
measure of program performance has several advan- 
tages including (1) the procedure is in place; 

(2) developmental costs have'.^already been absorbed; 

(3) it- constitutes ^a^-tompds-ite measure of client 
satisfaction which resptfruls to legislative and 
consumer advocacy concerns; and (4y the* data show 
some discrimination among closure statuses. 

The formula and d^ta source for this element is: 

. Numben of closed- clients 
surveyed who are satisfied . 
with their overall VR ^ 

experience : * -\, .(Closure Survey) 

Number of closed clients 

surveyed . • . ~ ■> ' ■ 
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The second data element attempts to gain a more 
detailed picture of client satisfaction with 
. specific key aspects of the overall VR process. y 
In particular,, the aspects isolated for inquiry * 
include questions about the client's counselor, 
the -physical restoration services received, the 
job training services received, and the job place- 
ment process. Consistent negative assessment in 
any one of these areas. Would be highly useful in 
guiding state .evaluations and providing substantive 
input to progranuiatic improvements. 

The formula and dat^ source for this element is: • ' ' 

a. Number of closed clients satis- 

fied with their- c ounselors en - n 

Number of "closed clients surveyed ( closure Survey 

b. Number of closed clients satis- 
fied with physical restoration 
services 



^Number of closed clients surveyed (Closure Survey; 



c. Number of closed clients satis- 
fied with job training, services 
Number of closed clients surveyed 

d. Number of closed clients satis T - 
fied with job placement services 
Number of closed clients surveyed 



(Closure Survey] 



(Closure Survey) 
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The third data element will assess the - percentage, 
of 26 closures judging services, received to have 
been instrumental in (a) securing their outcome 
situations, and (b) obtaining the skills necessary 
to ftjnction in tl^eir new positions. While not 
jjnequivocably objective, the client's assessment 
of whether he or she uses the skills and/or know- 
ledge gained from VR services is the closest 
approximation of the case. 



The formula and data source for this element is: 



/ Number of 26 closures judging 

the services they received to 
have been useful in obtaining 
their job/homemaker situation 
or in current performance 
Number of 26 closures surveyed 



(Closure Survey) 
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.35 1 *As you' can see, these eight standards cover a 

'" ' * wide- range of concerns and offer agency per- 

sonnel simple, yet comprehensive, measures of 
program performance. While the computations . 
necessary to arrive at each data element are 
( ■" ■ relatively straightforward, and involve only 
• V .simple arithmetic, it might be useful to quickly 

k walk -through the necessary calculations for one 
of the standards. 

. \ . ' ■ ' s • .. ■• 



^ , COMPUTING THE DATA ELEMENTS FOR STANDARD FIVE 

34 • As previously discussed, Standard Five states: 

THERE SHALL BE MAXIMUM PLACEMENT 'OF" REHAB IL- 
' ITATED CLIENTS INTO COMPETITIVE EMPLOYMENT. 
NON-COMPETITIVE CLOSURES SHALL REPRESENT AN 
IMPROVEMENT IN GAINFUL ACTIVITY FOR THE 
CLIENT. 



This Standard includes the following threes-data 
elements: 

• percent of 26 closures competitively employed 

percent of competitively employed 26 closures 
with' hourly earnings at or above the federal 
niinimum wage; and' 

• percent of non-competitivel^ employed 26 
closures showing improvement injunction and 
life status . 

TS - Because the actual information necessary to 

address the third data element has yet to- be 
incorporated in the RSA-300 system, our pre- 
sentation will focus only on the first two 
data elements for this standard. 

35 • The formula and data sources for the first data 

^ item-el^pnt* are as follows: 

/ Number%f competitively 

employ^|fej>6 closures (RSA-300) 

Number of 26 closures (RSA-300) 



Computing the *value of this data element requires 
three simple steps: 
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For the number of 26 closures, ^s^lect from the 
R-300s all clients in the Part 4.N.2 category" 
and count them. t 3 ' 

** • STEP TWO 

'«* ' ■ 

• Of these 26 closures selected, select, out! those 
whose work status (Part 4.1) is coded "1" or "3" 
for competitively employed and count them. 

** • -STEP THREE 

Divide the number determined in Step 2 by the 
4 number -in Step 1, and, because the quotient will- 
be a decimal number [i.e., .71), multiply by 100 
to determine the percent (i.e., 71%.). 

This percentage will then indicate the percentage ' 
of 26 closures who in fact were competitively 
' employed at the time their case. was closed. .Ideally, 

the higher this percentage the greater the success 
. of the program in meeting Standard 5. 

TS - Now, let us look, at the second data element. 

• The formula and data sources for the second data 
element are : 

* Number of competitively employed , „ 

^ 26 closures with hourly earnings 

at or above the federal minimum . 

wage ' ■ . ' : > ' ' - - (ftSA-"300 ) 

} Number of competitively employed (RSA-300) 

26 closures 

Computing the value of this data element also 
requires three simple steps. 

** • STEP ONE 

For the number of 26 closures competitively 
employed, select from the R-300s . ail clients 
in the Part 4.N.2 category and count all those 
coded "1" or "3". . . 



** 



STEP TWO 



Of these competitiA(ely-ehiployed closures, 
select out those who are indicated as earning 
an hourly. wage at or above the federal minimum 
wage . , . f . . . ' > ' 



Ho 



B-15 



I 



Time - • • . - 

from 'Overhead , '* ' s .. - 

Start Projection # . Points for Presentation 

■•**•. STEP THREE 



Divide the number determined in step 2 by the 
number in step 1, and, because the quotient -will 
be a decimal number, multiply by 100 to determine 
the percent. * 

This percentage will indicate the "percentage ' 
of those 26 closures who had been competitively 
employed and were earning an hourly wage at or. 
above the federal minimum level. Again, the 
higher the percentage, the greater thje success 
of the program in meeting^Standard 5. 

TS - We have now covered the specific instructions 
for computing each of the -data elements of the Per- 
formance Standards. There are two additional 
general instructions for computing the data elements. 



37 ADDITIONAL INSTRUCTIONS 
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First, the client Closure and Follow-up Surveys 
need to be merged with the individual clients 1 
RSA-300s". .:. 

• For the Follow-up Survey, this is necessary so 
that comparisons may be made between the client's 
situation at closure and at follow-up. 

• The Closure Survey data should be merged to 
provicie access to clients ' individual service 
records and personal characteristics. In this 
way, agencies can conduct policy-related analy- 
sis when problems in performance appear in sat- 
isfaction and service utility data elements. 

• Merging requires that a consistent identifier 
appear on both the RSA-300 and the. surveys such 
as client case number or Social Security number, 

* Finally, all calculations must be made using only 
valid. cases. This excludes cases for which the 
specific data item-is missing because: 

• the counselor was unable, to obtain the item for .„ 
entry on the RSA-300; 

\ 

• the client gave no response to the question on 
the survey; or 
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• the question was not appropriate to the client 1 
circumstances (such as assessing satisfaction 
of a service the client didn't receive). 



38 Restricting the data used to. valid cases only can 

make a substantial difference ^n the computation 
of data elements. In this example, the correct 
computation uses only 26s with valid earnings data 
at closure in^the denominator and makes a substan- 
tial difference in the answer. 



CONCLUDING COMMENTS 

• This concludes the formal presentation. As I 
stated in the be^inning^, all of the information 
presented today is included in the Trainee 
Handbook, Section B. 
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OBJECTIVES OF THE PRESENTATION 



{ 

Present rationale for each standard; 

. . ■ , v 

Outline rationale and data source for 
each data element; and 

Discuss how to compute data elements for 
a sample standard. 
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PERFORMANCE STANDARDS 



Coverage . 



••' Efficiency - 



Impact 
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Coverage 



STANDARD 1: 



Vocational Rehabilitation shall serve the maximum 
proportion of the/potentially eligible target 
population., subject to' the level of federal program 
funding and priorities among clients. 
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DATA ELEMENT 1(1): Clients Served per 100,000 Population 



Formula 



# served In a given year 

t 

State population (In 100,000s) 
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Data Sources 



RSA - 113 



U. S. Census Series "P-25 
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DATA ELEMENT 1(11): Percent Severely Disabled Served 



Formula 



# severely disabled served 
Total 1 # served 



In a given year 



Data Sources' 



RSA - 113 
RSA - 113 
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Cost- benefit 



STANDARD 2: The Vocational Rehabilitation program ..shall use 
*■ -resources in a cost-effective: manner arid show a 
positive return to society of investment in 
->* vocational rehabilitation of disabled clients.- 
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DATA ELEf'ENT 2(1): Expenditure -per Competitively Employed 26 Closure 



Formula 



Total agency expenditures 

# competitively employed 26 closures 



Data Sources 



RSA - 2 
RSA - 300 
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DATA ELEMENT 2(jJJ £xpenditure Per 26 Closures 
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Formula 



Total agency expenditures 
# 26 closures 



123 : 



Data Souirces 



RSA - 2 
RSA - 300 
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, \ Costs .< 

• Program costs 
••R&D costs 



• Client-borne costs 



Benefits 

Increased pdid earnings 
Change in output 
Reduction in public 
assistance 
Increased work 
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Rehabilitation Rate 



STANDARD 3: VR shall max ijri|ze the number and proportion of 

clients accepted for services, wrto are successfully 
% , rehabilitated/ subject to the meeting of other , ;i 
■ standards. : ^ 



13 
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ATA ELEMENT 5(1 ) ■ Percent 26 closures 



B (13) 



6. 



Formula 



# 26 closures 



# 26 + 28 + 30 closures 



Data Sources 



RSA - 113 

RSA -*11'3 
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flATA ELEMENT 5(11) : Annual Change in Number of 26 Closures 



Formula 



it of 26 closures In current year 



minus 

# of 26 closures In previous year 



13: 



Data Sources 



vRSA - 113 
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Economic Independence 



STANDARD 4: Rehabilitated clients shall evidence economic 
independence. 
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DATA ELEMENT 1(1); "Percent 26 Closures with Weekly Earnings tit nr. nhnvp 

Federal' "Rinlium Wooe ■•■ 4 v 



9 4 



Formula 



#26 closures with weekly earnings 
.1 eye 1 at/above the federal minimum wage 



1 26 closfjres 



I 



,4 



it 



V. 



* " i. 1 
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Data Sources 



■ 300 (Part y,2) ' 
Statistical Abstract of U.S. 

RSA ■ 300 1 



13 



DATA -ELEMENT W1L) ; Comparison of Earnings of Competitively Employed 
. - • * 26 Closures to Earnings of Employees In State . , 



/Formula 



Mean weekly earnings -of; competitively 
employed 

Hean weekly earnings of employees in 
the state 



.133 



Data Sources 



RSA - 300 (Part 4.1) 

U,S, Census Bureau 
Statistical Abstract 
of the 
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PAYROLL CHECK. 



7 



V 



Gainful Activity 



STANDARD 5: There shall be maximum placement of rehabilitated 

clients Into competitive employment. Noncompetitive 
closures shall represent an Improvement In gainful . 
activity for the client. 



s 
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DATA ELEMENT 5(1): Percent 26 Closures Competitively Employed 



' Formula 



# of competitively employed 26 closures 

■ — : r- — - 

, # of 26 closures ^ 



14 1 



Data Sources 



RSA - 300 - 
RSA - 300 
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DATA ELEMENT 5(11).: 



Percent Competitively Employed 26 Closures wl/fti Hourly 
Earnings at/above FederaJgEfyftimum Wage ■ 



Formula 

// of competitively employed 26 closures 
with hourly earnings at or above federal 
minimum wage 

# of competitively employed 26s 

v 



ft ■ 

Data Sources . • ; 

• RSA - 300 '■ [ 

• U.S. Census Bureau 
Stat 1st leaf Abstract 

> of the U.S. 

RSA - 300 : \ 



DATA ELEMENT 5(1 1 1 ) : Percent Noncompetitive ly Employed 26 Closures Showing 
; :■%["- Improvement In Gainful Activity 



... Formula ; 

if noncompetl tlve 26s w 1 th Improvement 
on LSI -FA I measure froip Plan to closure 

# noncompetitive 26 closures 



Data ^Sources 

RSA -\500V^ 
RSA - 30(5 /. 
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obligated Clients Shall Evidence Vocational Gains 



B (23) 



Formula 

Sum of closure earnings for 26 closures' 

■ minus 
sum of referral earnings for 26 closures 

# 26 closures 



14 u 



Data Sources 



Closure Survey 



• RSA - 300 
RSA - 300 
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DATA ELEMENT 6(11) : Changes In Functional Ability and Life Status 



(Changes In other statuses, and functioning ability, will be Included when 
appropriate measures become available.) \ 

... •*'• • ■»* 




Retention of Benefits 



STANDARD 7: Rehabilitated clients shall retain the benefits, 
of VR services. j , 

■* » 
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ATA ELEMENT 7(1) : Percent 26 Closures Retaining Earnings at Follow-up 



Formula 



H 26s with /earnings at closure who 
retained dr\ Increased earnings at 
follow-up 

# 26 closures with earnings at 
closure^ surveyed at follow-up 



Data Sources 

• RSA - 300 

• Follow-up Survey 



\ 
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DATA ELEMENT 7(11) ; Comparison of 26 Closures with Public Assistance as 
_ Primary Source of. Support at Closure and at Follow-up 



FormQla 

1 26s with public assistance as primary 
source oft support at fol low-up 



cuit; 



% 26s wKft public assistance as primary 
source of support at closure 



Data Sources 



• RSA r 300 

• Follow-up Survey 
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DATA ELEMENT 7(111) : Percent Noncompettttvely Employed 26 Closures Retaining 



Closure Skills at Follow-up. 



Formula V 

# noncompetitive 26 closures 
retaining LSI /FA I closure ski lis 

# noncompetitive 26 closures 
surveyed at follow-up , 



... 



Data Sources 



RSA - 300 
Follow-up Survey 
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CHent Satisfaction 



.A ... 

Clients shall be satisfied with the Vocational 
Rehabilitation^ Program, and rehabilitated clients 
shall appraise Vocational Rehabilitation services 
as useful in achieving and maintaining their 
vocational objectives. 
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DATA ELEMENT 8( 1 ) :^ Percent Closed Clients Satisfied with Overall VR Experience 



Formula 

it closed clients surveyed satisfied 
with overall VFt experience 

V closed clients surveyed 



Data Sources 



Closure Survey 
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DATA ELEMENT 8(11) ; Percent Closed Clients Satisfied with Specific Aspects of VR 



Formula 

# closed clients satisfied with their counselors; 

# closed clients satisfied with physical # 
restoration services; 

U closed clients satisfied with Job training services; 

# closed clients satisfied with Job placement services 

U closed clients surveyed. 



tf- 



Data Source 



Closure Survey 



1+*» J 
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DATA ELEMENT 8(111) : Percent 26 Closures Judging Services Received to have been 

Useful In Obtaining Their -Job/Homemaker Situation or In 
' Current Performance > j 



Formula 

U 26 closures judging services received 
as useful In obtaining their Job/ 
homemaker situation or in current 
performance 

# 26 closures surveyed 
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Data Source 



Closure Survey 
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1978 RECOMMENDED PROGRAM STANDARDS 



Performance Standards 



B (3 



STANDARD 1: 



STANDARD 2: 



STANDARD 3: 



STANDARD 4: 
STANDARD 5: 



STANDARD 6: 
STANDARD 7: 



STANDARD 8: 



VR shall serve the maximum proportion of the potentially eligible target 
population, subject to the level of federal program funding and 
priorities among clients. 

The VR program shall use resources in a cost-effective manner and show 
a positive return to society of investment in vocational rehabilitation 
of disabled clients. 

VR shall maximize the number and proportion of clients accepted for 
service who are successfully rehabilitated, subject to the meeting 
of other standards. 

Rehabilitated clients shall evidence increased economic independence. 

There shall be maximum placement of rehabilitated clients* into com- . 
petitive employment. Noncompetitive closures shall be in accordance 
with the IWRP goal and shall represent an improvement in gainful 
activity for the' client. 

Vocational gains shall be attributable to VR services. 

Rehabilitated clients shall retain the benefits .of VR services. 

Clients shall be sat is {f ed with the VR program, and rehabilitated 
clients shall appraise, VR services as useful in achieving and 
maintaining their vocational objectives. 
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STANDARD 5: There shal Y be maximum placement of rehabilitated clients into 
competitive employment. Noncompetitive closures shall represent 
an improvement in gainful activity for the client. 



' Data Element 5(i): Percent 26 closures competitively employed 



Data Element 5(ii): Percent of competitively employed J26 closures 

with hourly earnings at or above the federal 
minimum wage 



Data Element 5(iii): Percent of npncompetitively employed 26 

closures showing improvement in function 
and life status 



I*** * 
0 / 
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DATA ELEMENT 5(1)' 
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)rmula and Data Sources: # of competitively employed 26 closures (R - 300) 

# of 26 closures fc • (R - 300) 

✓ 



/Step One: Count all clients in Part 4.N.2 = # of 26 closures 



Step Two: ] Pull out and count all clients # of 

1 1 dent i fed in Step One who • = competitively 

were coded "1" or "3" employed closures 



Step Three: Step Two -s- Step One = Data Element 5(i) 



I 
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DATA ELEMENT 5(11). 



Formula and Data Sources) 



X 



§ ot, competitively employed 
26 closures with hourly earnings 
at or above federal minimum wage 

t of competitively employed 26s 



(RSA - 300) 



(RSA - 300) 



) 



Step One: 



Count all -clients In Part 4.N.2 
who were coded "1" or "3" 



= # of 26 closures 



Step Two: 




Pull out and count all of the 
clients Identified In Step One 
who had hourJy earnings at or 
above ,the federal minimum wage 



Step Two -f- Step One 



§ of competitively 
employed 26 closures 
with wages at or 
above the federal 
wage 



= Data Element 5(11) 



ADDITIONAL INSTRUCTIONS 



Merge client surveys with' RSA - 300 data 
using, a consistent client identifier. 




's* ■ 
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DATA ELEMENT : Percent 26 Closures with Weekly Earnings at or above 

Federal Minimum Wage' , • ^ 



• Using all 26 closures: 

i 400 (# earning weekly minimum wage) 
1,000 (all 26 closures) 

' • \ 

• Using only valid cases: 

400 (# earning weekly minimum wage) 

1 800 (# 26s with valld^eaVnlngs data 
at closure) 



40% 

/ 



= 50% 
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Vocational Rehabilitation Program 
Standards Evaluation System 
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Presentation Plans and Overheads-C: 
Administering the Closure and Follow-Up Survey 
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Points for Presentation 



AE&INISTERING THE CLOSURE AND ' FOLLOW-UP . 
SURVEYS 

• Does everyone- have a packet which looks^ 
^i^te this? • • ^ 

• All the information to be ^iscussed today 
is contained in this packet. Use it as a 
reference guide both during and after the 
presentation. 

TS - Today's presentation has two objectives: 



OBJECTIVES OF THE PRESENTATION (REVELATION) 

• First, to familiarize VR staff with the 
rationale for the Closure and Foljow-Up 
Surveys and the relation of tjie Surveys 
to, the entire rehabilitation process and 

• Second, to advise VR staff on the mechanics 
and administration of the surveys and 
possible ways to improve upon thm surveys 
for different states and district!!.. 



RATIONALE FOR CLOSURE AND FOLLOW-UP SURVEYS 



** • VR Performance Standards require that VR 
clients 



1) bje .satisfied with the VR services 
tlVey receive. (Standard 8). 



If'^' ^^m^^^/^^ finc * W services useful <dn obtaining 
% ^ ^ and performing jobs (Standard 8) 

and 



retain 'benefits of the VR p; 
(Standard 7)? . • 



am 



' : 't*-;'C*:i'^^". • Th e best way for VR or states to determine* 
^ 4^ / .v-/A*' e whether these requirements have been met 

1^) actually question VR Qlients. 



vtS^^^^v^^- • Hoi 



However; consider these requirements in 
terips of the rehabilitation process. 



erJc ^ 



Points far Presentation - 

• Determining whether the first two require- 
ments had be'enSjmet could, occur at case * 
closure. Whether or fpt clients retained 
the benefits, hpwever^^ould only be 
determined sometime aftfer case closure, . 
when the client's situation could be 
followed up upon. 

• Consequently, two surveys were developed: 
the CLOSURE SURVEY and the FOLLOW-UP SURVEY. 

# " 1 

TS Let's examine the Closure Survey first. , 



CLOSURE SURVE^ PAGE 1 

• The Closure Survey is designed to be self- 
. explanatory iaild is to be completed by the 

client 3 at case, -closure. 

It consists of nine questions and each .is 
an essential source of data for computing' 
specific performance standard data elements. 

• For instance, Question 1 is used i%com- 
puting .data element 8(i); percent of closed. 7 
'clients satisfied with their overall VR 
experience^ 

CLOSURE SURVEY : PAGE 2 - 

• Questions 5 &id 6 are used in computing data 
element 8(ii)Vpercent of clos^ clients 
receiving job training" services*; and who 

are satisfied with those services . 

• Again, each question helps measure total 
agency performance and provides a .closer 
look at district and even individual coun- i 
sel or performance. More specific detail 

' of how each question pertains to the v 
performance standard data elements in in 
your Trainee^Handbook. 

r Each question can also help identify 4<t 
strengths and/weaknesses in specific^ 
service areas or across different 
client groups. # 
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V; r • ^ 

> 6 FOLLOW-UP SURVEY: PAGE 1 

• The major purpose of the follow-up survey 
is to determine the extent to which benefits 
Y" ' gained by ,VR clients have had a long-term ( 

r "' effect 

t i' 

* i ;"' • The follow Vup survey is also designed' as a 

VV.' mail-back questionnaire to be completed by 




^ r ■■^Tt& formfe^^jclients one 'year after case closure,.' 

• It copgxkts c o^d^^qti(Sstions which ask 
cJj^t^^touteHS^eir *work status, eatings 

'-^^ r s&%$^ ; >'- each question provides . useful informa- 
^^»8n^ifp VR and state agencies and are used 




TV 
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iif computing specific data elements. * 

" " f . : v ^-r . ' ■ 

. •v^f or .example, Question 2 is used in computing 
; * H^Mata element 7(ii^; primary source of sup-' 

) ^ r vport for clients ' ' s 

* . * ■ .J ' ; v • 
' - FOLLOW-UP SURVEY : PAGE 2 

i ^ • If VR collects life status and functional 

' j/ f s . ability information, additional questions 

j. can be added' to track gain^in these 

areas during follow-up. * 

- # u ^Thrdt^h^ the Follow-Up Surveys, in conjunction 

■ a . ; .#ith "data about individual client character- 

istics, VR can learn much about the long-term 
. s impact of the program's services on different 

r t v types of clients, as well as how that impact 

*. 4 might vary across districts or counselors. 

x 8 ADMINISTERING THE SURVEYS iR<V 

• ■; . ^jf.- , " 

^ • There V^te five major activities in admin- 

istering the surveys: 

1) selec*J|ig the sample; ^ 

2) developing the si^ryey in,strument; 

* 3) distributing the survey^. k. ■ 

4) collecting the survey; and 

5) editing the data! v 
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• Each of these activ±€ies is essential 
to collecting high-quality data. 



* 5 



LINKS IN THE SURVEY/DATA COLLECTION CHAIN 

c 

• View each task as links 3 in-a chain. If 
any of the" links are weak, the whole chain 
is weakened. Each task, therefore, re- 
quires planning and^ ethought. 

TS - Let's look more closely at each of these 
tasks. 



iO SAMPLE SELECTION 



• In one sense, the best HSay to prevent 
sampling biases or other sampling problems 
would be to distribute .Closure and Fol low- 
Up Surveys to all clients. v 

• Y^t this is not only impractical in terms 
of time and costs, it is unnecessary as 
well. /jfe „ I- 

• . For the purposes of these surveys, a ran- 

domly selected sample of appropriate size 
will provide adequate data to respond to 
the VR Performance Standards as well as 
provide VR with a valid base for collecting 
inforftation about any dtjier aspects of the 
program that the -agency may choose j^tg include 
in the surveys. 

• Many. questions arise when considering the, 
selection" of the sample. [' 

• First, WHO SHOULD BE INCLUDED*!^ THE S 

• ■■ * ■ & 

• "For the Closure Survey, clients Whose cases 

• were closed after receiving VR services 
should be included: 



1) 26 closures, 

2) J28 -closures, 

V 3) 30 Closures. 



Time 
from 
Start 
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Overhead . 
Projection 
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<. 

• For the Fol low-Up Survey, benefit reten- 
tion is relevant only for rehabilitated 
clients. The sample should only include: 

1) 26 closures. ^ 

• The second question which arises is 

HOW MANY CLIENTS ARE NEEDED FOR THE SAMPLE? 

• For the Closure Survey, 500 clients should 
be included in the sample each year. This 
sample should include: 

1) 300 - 26 closures, or 25 per month . 

2) 100 -.28 closures, or 8-9per montih 

3) 100 -v 30 closures. or'S-Qper month 

• The Fol low-Up Purvey sample should include: 
1) 200 - 26 closures, pr 16-17 per month 

• ■ The 200 clients sampled for follow-up need 
'-not be part of the. Closure Survey sample, 

• For both surveys, clients should be sampled 
-throughout the year on a monthly basis. 



THE MONTHLY SAMPLING PLAN 



The best way to dto this is to develop a 
monthly sampling plan, which involves three 
steps. Using ah example will help illus- 
trate the process: 

1) At the end of elach month, record the 
number, of case closures for each 
closure status during the month.^ As 
you can see, in the example, we had 
250 Status 26 closures, 99 Status 28- 

• closures, and 78 Status 30 closures. 

2) Divide this number by the number of 
client^ required for the sample in 
each status. As mentioned earlier, 
25 StatusS|;26 closures, nine Status > 
28 closures, and nine Status 
closures are required. 



Time 

from Overhead 
Start Projection # 

- *, • 
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3) The result of this division represents 
the frequency with which filed cases 
should be selected for inclusion in * 
the sample. For the example^, .in... this case 
every tenth 26. closure case should be Se- 
lected. Similarly^ -■^^^^l:eyehth : ;?8. r , 
closure and every ninth $0 closure should 
be selected for the sample. 

A new monthly saijipling plan must be made 
each month. To facilitate this process, 
sampling forms for an entire year are 
included in the Trainee Handbook. 

If you 4on f t have a computerized tracking 
system, the preparation of the Monthly 
Sampling Plan and the distribution of the 
Closure and Follow-Up Surveys can be eased * 
by organizing case closure files according 
to closure status on a monthly basis: 



SURVEY INSTRUMENT DEVELOPMENT 

• The second liYik- in the suirvey chain is the 
development of the survey instrument . .> 

• A$ both the Closure and Follow-Up Surveys 
nave already been formally prepared, this 
link is , complete. 

• However, some states may desire to take 
advantage of this survey contact with clients 
to ask additional questions which respond 

to the state's own planning and evaluation 
. ' needs, f 



Advice on how to prepare questions for 
addition ±£ the surveys is included in 

the Trainac Handbook. / * 

* .* ■•* • 

Instructions for proper printing of the 
^surveys is also included^ 

J^y,. t .... • . 

JRemeinbe^ ally states miist use the stan- 
dardized .-survey, questions exaitly as they 
'Ore px;^arj0<^^y^toy a ^d -their own 
questions at t$£.£hd of th£ surypy. 



Overhead ' 
Projection # 
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13 SURVEY DISTRIBUTION 



; <)ftte' the questionnaire has been prepared 

for distribution, there are still three 
;J ;T - ; tasks Jtfhicli viiust be completed. 

• First, a cover letter 'should be prepared 
on agency letterhead. 

• The cover letter should introduce the survey 
and motivate the client to immediately com- 
plete and return the survey.' It should 'be 
short and concise, yet should provide adequate 
information. 

• Sample cover letters for both the Closure / 
and Fol low-Up Surveys are included in the 
Trainee Handbook. Use them to hJlp prepare 
your own agency's letter. 

• Two other points about the cover letter 
should be stressed: 

1) Be sfare to date the letter. Doing 

so gives the letter .an air of greater 
importance and gives the respondent 
a frame of reference of elapsed time * 

2) It is best to type the name, Strafress, 
and salutation ontp^each letter so it 
appears- more personalized than a 
typical form letter. \ _ 

• Secondly, a system ^for keeping track of surveys 
needs to be established. ^ I A 

• Use tfre Survey Control Shefets which are 
provided in the Trainee Handbook to help 
set up this tracking system. 

TS -.Let's take, a closer look at the Survey 
s " Control Sheet. 



V 
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SURVEY CONTROL SHEET : TOP HALF 

• 'This is the top half of the Survey Control 
Sheet, It contains basic information 
about the client including the case number, 
Social Security number, closure date, and 
name, address and phone number. 



15 



SURVEY CONTROL SHEET: BOTTOM HALF 



The bottom half of the Survey Control Sheet 
has information more pertinent to the Clo- 
sure and Fol low-Up Surveys. 

The survey mailing date should be filled In 
on Item 1 along with an acknowledgement 
of whom it was sent by. 

A 

That's all you need to do to begin with; 
we* 11 come back to the other parts of the ^ 
Survey Control Sheet in the next section. " 
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SURVEY DISTRIBUTION 



6 

9 



( 



So, in review, the first two steps of Survey 
Distribution are to prepare a cover letter 
arid a Survey Control Sfyeet for each client. 

The third aspect of survey distribution is 
to mail the cover letters and surveys. i 

Be sure to enclose t return envelope, 
preferably one that is postpaid. 

Try to mail out the surveys at the beginning, 
of the week' — it avoids weekfend buildup' 
^nd increases the probability of a response. 

Lastly, don't forjget to put the mailing 
date in the appropriate space on the Survey 
Control Sheet. 
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SURVEY COLLECTION 



;\Most people who answer questionnaires do 
^TV'SO almost immediately after they receive them. 
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When people do not return a questionnaire, 
it could be for a host of reasons: 

1) it never got to the person it was 
addressed to; 

2) it was discarded as "junk" mail; 

3) it was filled out but, for Vqe reason 
or another, it was never returned; 

4) the client was unable to fill out the 
quest iojpngire^because of his or her 
disability 

Tliere is, however, a four-step process* which 
can be used to maximize the response rate: 

1) Two weeks after the initial distribution/ 

send * reminder postcard to clients c f" s \ 
to' jog their memories and influ^nce^ their ^ 
priorities. Be sure to stress the im- 
portance of their response, offer a 
.replacement questionnaire, ^nd thank 
«early respondents [in case the reminder 
^hais crossed the response in the mail) . 
A sample postcard reminder is included 
in the 'Trainee Handbook - 



Some states may choose to use a computerized 
flagging system to keep track of when re- 
minders and- response prods should be made. 
However, this can also be accomplished quite 
simply through the use of the survey con- 
trol sheets. When questionnaires .are re- . 
turned by clients, the date is recorded y 
and the control sheet is pulled from thfe 
pending file. All those remaining in the 
current month's pending file at the end of 
the two-,week period would receive a two- 
week reminder. 
/ 

One week after the postcard reminder, 
send another questionnaire and another 
cover letter^ This cover letter should 
combine elements of the first cover 
letter and the postcard reminder. A 
sample of this cover letter is also included 
in the Trainee Handbook-. 
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3) If ahother w6ek passes and the survey 
hasn't been returned, giveythe client a 
telephone call to gently urge a response. 
The focus of the phone call should be to 
determine the cause for the delay and 
whether or not the client is willing to 
participate, 

4) If cooperation has been assured over 

the phone, yet no survey has been returned, 
the agency may elect to mail a third 
questionnaire -or to complete the survey 
through a telphbne call or' a. personal 
interview with the client. This "last 
try" step is important in ensuring 
that the sample includes the most - 
severely disabled who may require 
special assistance. If no survey 
has been completed and. received 
six weeks after the initial mailing, 
consider the survey a nonresponse. 



SURVEY CONTROL SHEET: BOTTOM. HALF 

• If you need to take any of these steps to 
maximize the response rate of tlie surveys, 
be sure to record when you took those steps 
on the Survey rControl Sheet* 



DATA PREPARATION 

• The TjetffyTTi of the surveys to the VR agency 
is n<^,;the end of the survey/dat& collection 

, process". One other task must be undertaken 
before the surveys are ready for analysis. 

• It is important that afi completed question- 
naires be edited to ensure that instructions 
were properly followe^, the answers afe 

^appropriate to the question, and that the 
correct number of responses were given. 

• ' The agency should strive to have only one 

or two survrey editors, as this will allow " 
greater consistency and will allow problems 
to be recognized earlier. 



.Time ; : " " 4 
froft .JJyerhead .* 
Start .I*rojecfeion # ^ 
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""SjbpSURE AND FOLLOW-UP SURVEY REVIEW 

Jn review, both the Closure and Follow-Up 
Surveys are of critical importance in 
assessing the extent to which VR services 
have benefited clients on both short- and 
long-term bases, ^ 



:J 
J 



There are five steps in administering the 
surveys which can be facilitated by using 
forms and samples in your packets: 

1) select the sample; 

2) develop the survey instrument;. 

3) distribute the survey; 

4) cffllject the survey; and 
55 ^eSit the data. 
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Vocational Rehabilitation Program 
Standards Evaluation System 



Trainee Handbook-C: 
Adtfiinistering the Closure and FolfoW-Op Sulteys 
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v ' OBJECTIVES OF PRESENTATION 

i to famfllarlze VR staff *wlth the rationale 
of the llosure and Fol low-Up Surveys 

i to advise VR staff on how to administer the 
surveys. 
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RATIONALE FOR CLOSURE AND FOLLOW-UP SURVEYS 



VR REQUIREMENTS: 



1 



Clients 
Satisfied 
(Standard 8) 



Services useful 
for Jobs 
(Standard 8) 



TIMING: 



INSTRUMENT: 



Closure 
Survey 



Benefits 
Retained . 
(Standard/) 



. ; 


' / * 

/.... ^ 

/ " 'flBF YEAR V 


REHAB PROCESS 


■ ' - $• 

I ' 





4 * . A 



r v 



4* 



Survey 



1. 



Are you satisfied with your overall experience 
with the rehabilitation Program? I PLEASE CHECK 

Inej 



i. Yes 



2. No 



1 3$&t$u re.- 6r^j^0i n 1 on 




i 2;\f Are you satisfied with your counselor's per- 

| formance (that ||&|dld he/she do a good job 

\ for you)'? (PLEAsFcttECK ONEl V* 

] i. Yes 

i 2. No 

! " 9. Npf sufV or no opinion 

[ 3s Did your ^counselor arrange- for yoy. to have, ; 

i physical restoration services^ such as meifca] 

j ' ! it reagent, physical thefcgpy,, artificial ffijjbs, 

i eye^losses/ dentures/ftearlng aids, etc.? * 

I f PLEASE CHECK ONE] 

i i'J Yes • 



*>. 'i gfaj&'t remember 




■/ 




(Column f) * . ,| 

' ] 4. If YES/ ore you satfetfed with these' ser- 

■ |. vices? [PLEASE CHECf ONE! . 

Gi8) i __i. Yes ,; ^ ... V V 

• _ 2. No ••; . -C't 

V ! * : Not sure . or no opinion . / r /, 

; . 1 5. Did y^ir counselor arrange, for you to have 

! Job training? [PLEASE' cfiEQK ONE] 4 

H9f K i • -Yes 

"v :': 2 - no 

.> : 1 •• 9 - I don ' t- remember ... >. 



• 6*. If-YES/dre you sdtisfl'# with the kind .Qf. 



training you received?- Jf PLEASE CHEC 
(20)* * . : i. Yes*^ ••••?• -:*r- : 

i :,. 2 - NO r% -*< ■ * 

! « 9> Not sure .or. no opinion .* • 



. . .or 




I 

J 



! 7, Did your fcouq$eloj; hel|iyou look.for a J ob? . 
fPLEASE CUti&pUtr ^ 



(2i) ! _ K Yes 



w i _ 2 - NO ^ & 

1 _ 9- Tdon't remember 



i 

1 8* If y£s, are you satisfied with the hel 



received? [PLEASLXHECK ONE] 



(22) i li^Yes Sk , N 

- « 



CLIENT FOLLOW-UP SURVEY; PAGE I 



- VR CLIENT FOLLOW-UP SURVEY 



1. 




/Jjftlch of the following statements best de- 
scribes your' present work situation? [PLEASE 
CHE££ ONLY ONE! 

;■ i. I earn a wage or salary, either 
regular Job or from self-emplo 

' 2.1 earn a wage or salary In a shel 

workshop or Business Enterprise Pro- 
gram (BEP) 
I am a homemaker 

I work InVfamliy farm or business 
without pay 

I am not working at present. 
Other (explain);" s •? 



3. 
4. 

i» 

5. 
6. 



■ (16-19) 



2. How much total Income, If any/ did iou and; 
1 • jgu/. dependents receive last month from alt V 
Purees of pub Uc welfare? [PLEASE .CH^K . 
.Y ONE AND, FILL IN TH& SPACE] 

We received $ last month 

None 



■ ^ A 



I tionZt remember 3 




c© 



18x 



ERIC 



[Column- #) 



(20-21) 



6*' 



(22-26) 



CLIENT FOLLOW-UP SURVEY : PAGE 2 



3. What were your totgl earnings last week (from 
a Job> self -employment, sheltered workshop, v 
of Business Enterprise. Program (BEP)? 
_ I earned $ last week 

I am working but I don't receive a 
wage or salary a 

I am not working 8 

« r don't know 3 



a. What was your Income last month from private^ 
. sources other An the earnings reported fftf^ 
Question 3 '(for%xample, from rents, .dlvl- 
^ dengjv or private Insurance)? 

W I received $ last month 

V. None 3 'k 

I .don't remember 3 . 

_ j_ ^ _ _ • r _ C_'4Sfei : . l 'i' 

S.\ ( I tenis^S^ssing -functional q^ijityj§d IJfe 
status; items to^be determined through pr6- _ 
test of : .the Life. Functioning Index (LFI ) . 



.ft. 
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ADM I NJSTERING THE SURVEYS 



i V 



1) 
2) 

."I 

3) 
5) 



Selecting the sample ."• 
Developing the survey instrument 
Districting the survey 
Collect J rig the survey '' ; 



Editing the data 



.' A. 



j 
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PUNNING 




i-| ' SAMPLE 
V SELECTION . 

. (4 * 



'A 




SAMPLE SELECTION 



I, WHOM JO INCLUDE: A, Closure Survey 

% i 26, closures 

' . ; • 28 closufeTp 

■ • 30 closure! 

B, Fol low-Dp Survey 
i 26 closures 



7>V 



2, 




A, Closure Survey 

ji (J 

• 300 - 26 closures/year or 25/month 

• 100 - 28 Jjsures/pr or 8-9/month 
'< • 100 - 30 closures/yetir or 8-9/month 



v 



B, 



. i 200 - 26 closures/year or 17/month 

■ ,/ 

W SHOULD MONTHLY ' , ,' „ ' 
SAMPLt BE SELECTED? • Develop Monthly Sampling Plan 
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MONTHLY SAMPLIN&^ a 




Status 


-« _ — . 

• Month 


Actual ¥ t 
of JCoses 


© & 

Monthly 
* Sample 
^Slze 


• © 

Select 
Every 


26 
28 
i30 


250 
99 . 
78 . 


* :- 9 . 

f-9 ' 


10th 
11th 
£th 



■JBBT 



ft 
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Purvey distribution 



STEP 1 : Prepare the cover letter,, 



STEP 2: , Set up the recordkeeping system. 



STEP 3: Package and mail out the survey. 



• . ■ SURVEY CONTROL SHEET - TOP HALF 

' •> / < 

• 'I 

ipstlonnalre No, ' 

lumber on Survey Control Sheet "and 
lestlonnalre must correspond). 

v. ■ . 

VOCATIONAL REHABILITATION CLOSURE SURVEY 
T, Survey Control Sheet - 
To Be Completed by Agency Personnel Only 

Agency and Client Identification 

Case. No. I I I I I I I I I I I 2. Closure Date / / 
Social Security No, / / / / - / /•/' /-/ / / / / 
Client's Name ■ ■ ■ 



Client's Address ' : . , ., 

-street name ana number T Apt, no, 

"cTfy ' ~~ ~ slaTe zip coae 

Client's Telephone No, ( ) - I 



survey; control sheet ■ . BOTTOM half 




II, Survey Control Information- '(Check the boxes that apply and give the dates) 

1, Initial Questibnnaite Distributed --Date /_^/_/_/ v 

How: by counselor M > " • „ . ,• 

by district office? /_/ ' 1 . „ 

by* central office? U , . 

Response Prods : { . .. ;. ' *. 

2, Reminder postcard * . • /_/ • Date/,./ / / ; 

3, Second survey form • - U Date LLL/ 
X Telephone follow-up . "jU \ Date / / / / 



5, Other: ^_ _ /_/ ■ Date / / / / 

6, Classified as nonresponsr^ /_/ Date /■■/'/ 7 

7, Questionnaire Completed? /__/ ; Date / / / / 
How: by mail? I_J 

by telephone? /_/ . Interviewer's name: ; ■ ' 



in person? ' / / 



1 



SURVEY DISTRIBUTION 




STEP T : Prepare the cover letter, 



j 1 



STEP 2: Set up the recordkeeping sVstem. , 



STEP 3: 'Package and mall out the survey. 



SURVEY COLLECTION 



STEP h Two-week reminder, 



Step 2: Second questionnaire (third week) 



Step 3: Telephonb reminder (fourth week), 



Step 4: Last try/ 



SURVEY CONTROL SHEET - BOTTOM HALF • 



r 



4 ' 



II, Survey Control Information (Check the .boxes' that apoly and give the dates)' 

. 1, Initial Questionnaire Distributed — Date 
How: by counselor l_J 
. by district office? hj 
by central office? U 

Response Prods : 

i Reminder postcard '. . U Date / '/._/,./ 

3, Second survey form /J Date/./,././ 

t|. .Telephone follow-up ' U Date MX./ 

5. Other: ; ■ ' - — __ /_/ Date LLL.I 

6, Classified as nonresponse U Dote I 

,7, Questionnaire Completed? L/' . Date /"/ / / " , 
How: byjuall? /_/'/,'... 

by- telephone? / / Interviewer's name: . r 

i in person? U ' ■ - 



cm 



:losure and follow-up survey -review 

i Surveys of great importance in assessing VR services • 

i Five steps in administering: 

1) Select the sample 

2) Develop the survey instrument 

3) * Distribute the survey . 

4) Collect the. survey . 

5) Edit the data • . * 
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The Five Procedural Standards 
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Time 

From Overhead * " 

Start 'Projection # Points for Presentation v? 



1 . THE FIVE PROCEDURAL STANDARDS 



• " Does qveryone have a packet which looks like 
this? ' 



• All the information to be discussed today is 
contained in this packet. Use it as a refer- 
ence guide both during and after the present- 
ation. ' ^-w/^ 

• x " . ■ ^ 

TS - Today's presentation has two objectives:. 

'2 OBJECTIVES OF THE PRESENTATION ' 

' v • • To familiarize VR staff with the five Procedural 
' Standards. , 

■ *t£ ■ 

• To identify the specific data elements and 

' '} \ ■ * 4 sources-for these elements necessary far -assess* 

*- ing an agency's performance on each Procedural \ 

Standards 

3 * OVERVIEW OF PROCEDURAL STANDARDS 

■ ' „ • VR Procedural Standards are concerned with: - 

** l) Process: Is the information collected 

\ ' ' on clients valid, reliable, 

accurate, and complete? 
Are services provided %n a 
timely manner? 

** ' 2) Compliance: Are eligibility decisions 

made in accordance with the 
laws and regulations? 

** 3) Goal. Plan- Are realistic goals developed 

ning: for VR clients? Do goals set 

correspond to client out- . 
, comes? 

4 VR PROCEDURAL STANDARDS 

The five "Procedural Standards thus evaluate 
procedures at different steps in the rehabili- ' 
tation process: at intake, eligibility deter- 
\ m mination, plan envelopment, service delivery 

and outcome. 6 

TS - Now -let's take a closer look at each ofi. 
the five Procedural Standards. 
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STANDARD 9: R-300 VALIDITY' 

* 

INFORMATION COLLECTED ON CLIENTS BY 
THE R-300 AND ALL DATA RH^ORTING 
SYSTEMS USED BY RSA SHALL BE VALID', 
RELIABLE, ACCURATE, AND COMPLETE. 

This standard ensures that state agencies main- 
tain acceptable levels of accuracy, validity, and 
reliability in the reporting of the R-300,. £ts well a: 
other, data. By using a case review process, an 
accuracy check is provided between the case folder 
information, the R-300 itself, and any computer out- 
put listings of R-300 items selected for review. 
These items include a' range of demographic data, i 
service data, outcome data, and cost data. k 

v • 
TS - Now, let us move along to the next Procedural 
Standard. . 



STANDARD 10: ELIGIBILITY . 

• ■ ELIGIBILITY DECISIONS SHALL BE BASED # 
ON ACCURAtE AND SUFFICIENT DIAGNOSTIC 
INFORMATION, AND VR SHALL CONTINUALLY 
. REVIEW AND EVALUATE ELIGIBILITY DE- 
CISIONS TO ENSURE THAT DECISIONS ARE ' 
BEING 4 MADE IN ACCORDANCE WITH LAWS 
AND REGULATIONS. % 

This standard concerns the determination of those 
applicants who wiir receivek^services . Such eligi- 
bility decisions must: - 

* v ■ - . 

• comply with the legislative mandate, and 

• ensure cost-effectiveness. ^ 

In short, this standard focuses on ensuring that 
clients who are not eligible for VR services are 
not accepted, while. those who are eligible. are pro- 
vided with services. The case review ^system will 
be used to monitor decisions made in compliance with 
this standard. 

While monitoring and review of eligibility decisions 
by supervising counselors or managers^ will provide a 
check on that , determination, the actual procedures 
utilized in providing this supervision will not -be 
monitored. Consequently, states will be allowed \o 
retain flexibility im establishing their monitors/fig 
practices. • \ - { 



Points for Presentation 



TS - Let- lis loolc the next Procedural Standard. 

STANDARD IT: TIMELINESS . ' " 

VR SHALL ENSURE THAT ELIGIBILITY 
DECISIONS AND CLIENT MOVEMENT THROUGH 
THE VR PROCESS OCCUR ^IN A TIMELY 
MANNER APPROPRIATE TO THE NEEDS AND 
CAPABILITIES OF THE CLIENTS. 

This standard seeks to avoid delays 'in the VR pro- 
cess which may hinder or impede the successful 
rehabilitation of the client. It requires that 
each state have"a monitoring system to 'flag 1 those 
cases remaining in statuses for an untimely period, 
and provides a process for evaluating each such un- 
due delay. The reasons for this system are twofold; 

• First, a client's attitude toward the usefulness 
of participating in VR are formed by his or her 
perception of the' VR treatment , a perception 
often shaped by the speed with which his or her 
case is handled; and # 

• Second, a correlation has been found between VR 
timeliness and client outcomes. 

A specific timeliness assessment instrument has been 
developed and incorporated into, the Cas;e Review 
instrument which provides a reviewer with: 

• ( A r notation mechanism for determining whether a 
delay has occurred in terms of the time it takes 
to complete various necessary activities -in a 
case such as the eligibility decision; 

• An assessment of the reasons for such time v 
lapses; and \ 

• A notation of whether a case 4 was handled with 
'undue speed. ' 

The advantages of this system are that: 

• It allows for a flexible interpretation .of 

, ' whether a case was handled in a timely manner 
by avoiding rigid criteria regarding the most 
appropriate time frame for various case activ- 
ities; and 
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• It ensures greater reliability than other sub- 
' - ' tjective measurement systems by requiring' the 
^ ; " . reviewer first to determine if. a delay did occiir 

and then to determine the reason for a given 
v > delay (i.a., agency failure, client motivation, 

, other agency actions r etc.) j 

TS - '.Let us move on' to the fourth Procedural Standard. 

11 STANDARD 12: IWRP " 

. ' VR SHALL PROVIDE AN .INDIVIDUALIZED 

• "WRITTEN REHABILITATION PROGRAM FOR 

■ - EACH. APPLICABLE CLIENT, AND VR- AND 
- c- 1 *' THE CLIENT SHALL BE ACCOUNTABLE TO 

• . . . ; . EACH OTHER FOR COMPLYING WITH THIS 

. AGREEMENT . ■ 

12 Several aspects of the Individualized Written Rehab- 
ilitation Prpgram are addressed by. this standard. 
These include: 1 - 

that an IWRP be fully developed for each eligible 
VR client; / 

• that the plan ensure, th£ protection of clients 1 
' rights; 

J ♦ 

^ w s v - • that the client and counselor work together in 

. 1 developing all goals and service plans; 

i ' 

• that the client and counselor share responsib- 
ility for follow- through and the annual review 
of the progress and appropriateness of the 
Agreement; and 

< 

• that' the handling of any plan revisions are 
• timely and appropriate. 

This standard will ensure compliance with the legis- 
latiJHfcintent of the IWRP, which in turn has been 
1 , posiflpely associated With successful VR process 

OUtCOMS. 

TS - And now, let us turn to the final Procedural 
-Standard. 
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STANDARD 13 : GOAL PLANNING 

, T •■ COUNSELORS SHALL MAKE AN EFFORT TO «. 

v . SET REALISTIC GOALS FOR CLIENTS. 

. ; COMPREHENSIVE CONSIDERATION MUST BE 

GIVEN TO ALL FACTORS IN DEVELOPING 
APPROPRIATE VOCATIONAL GOALS .SUCH . 
THAT THERE. IS A MAXIMUM OF CORRESPON- 
DENCE BETWEEN GOALS AND, OUTCOMES: 
"COMPETITIVE GOALS SHOULD HAVE COM- 
PETITIVE OUTCOMES Atft) NON-COMPETITIVE 
GOALS SHOULD HAVE NON-COMPETITIVE 
OUTCOMES. / _ 

This standard concerns the setting of "realistic 11 
goals for VR clients, consistent with their capa- 
bilities and abilities, whether this means setting 
competitive employment goals or sheltered or non- 
competitive employment goals. It addresses the 
issue of noncompetitive closure categories being 
instigated simply to salvage "successes 11 for clients 
unafble to meet their planned competitive goals. By 
allowing flexibility in goal planning, the' standard 
neither hinders clients from obtaining competitive 
closures, nor clg^sifies them as "unsuccessful 11 
should they have a competitive goal but a noncompet- 
itive oiltcome. 

DATA ELEMENTS FOR STANDARD 13 

As the four data elements for this particular standard 
indicate, the standard's objective is to investigate 
how counselors can be more effective ijj/^the task 
o£ "fitting" clients'* potentials to feasible outcomes. 
In this way, the standatd is used appropriately to } 
facilitate effective goal planning . All four data 
elements are included in. the R-300 system. 

COMPUTING THE PROCEDURAL STANDARDS DATA ELEMENTS 

• The data elements for the Procedural Standards 
consist, for the most part, of individual inform 
nation items pertaining to specific aspects of 
the. standard in question. These information 
items will be presented in terms of a series of 
'•percentage achieved scores." With this, program 
managers will be able to see the extent to which 
an agency is incompliance in terms of a number 
of separate indicators/ This will allow .program 
managers to pinpoint specific problems occurring 
in the agency's case-handling and data-recording, 
processes. * 
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• \The one exception to the "percentage achieved" 
Method occurs on Standard 13. As noted in the 
discussion of that standard, its data elements 
cohsist of four similar ratios, each of which 
compare clients 1 IWRP goals to their ultimate 
outcomes. 
\ 

CONCLUDING COMMENTS 

This concludes our formal presentation of the 
Procedural Standards. As we mentioned at the 
beginning, allvof the information presented 
today is included in the Trainee Handbook, 
Section D. \^ 
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OBJECTIVES OF THE PRESENTATION 

~ r 

Provide an overview of the five procedural 
standards; and - 



Identify specific data elements and sources 
for each standard, 



PROCEDURAL STANDARDS ' 

Process - information accurate? Services timely? 

V ; 
■ \ i 

Compliance - regulations followed? 

« 

Goal Planning -goaLs realistic? Achieved? , 

* » 
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PROCEDURAL STANDARDS 



Goal Planning 




Information 
Collected 
on Clients' 




« 4 




Timeliness of Services 
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RSA-300 VALIDITY 

^ 

ensures accurate; valid, reliable and complete data; 

us^es case review to check accuracy between case folder 
and RSA-300; ' 

•at 

Items checked Include: 

- demographic data, 

- service data, 

- outcome data, ' 

- cost data. - ■ • . • 



ELIGIBILITY 

; STANDARD 10: Eligibility decisions shall be based on accurate and < 
* sufficient diagnostic Information, and VR shall con- 
tinually review and evaluate eligibility decisions to 
■ • "ensure that decisions are being made in accordance with 

" laws and regulations/ ' ' < 




ELIGIBILITY ' 

El Iglbl I itvcd^cisions must: 

- comply with the Legislative mandate , bod 

- assure cost-effectiveness. 

Case review will be used to monitor this standard. 

Procedures used for supervising review of eligi- 
bility decisions will not be monitored. 






TIMELINESS . ,. 

TANDARB 11: VR shall ensure that eligibility decisions and client 
movement -through the Vr process occur Jn a timely, 
manner appropriate to the needs and capabilities of v 
the clients. • 
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TIMELINESS 



Timeliness of service delivery can affect both client 
satisfaction and service outcomes; 



• a timeliness assessment instrument is included in 
* the case review. 



5 ' • A^(Bs^ht'!^pGedure;anows^f<)>^a flexible 'interpretation 
taking into account specific aspects of each case. 
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IWRP 



VR shall provide an individualized written rehabil- 
itation program for, each "applicable client,'. aha" VR 
and the client shall be accountable to each ot^her for 
complying with this agreement. ^ 
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GOAL PLANNING 



STANDARD 13: Counselors shall make an effort to set realistic goals 
• *- for clients, Comprehensive consideration must be given 
to all factors In developing appropriate vocational , 
.. goals such that there Is maximum correspondence between 
goals and outcomes: competitive goals should hove > 
;e competitive outcomes and noncompetitive goals should 
J< "', have noncompetitive outcomes. 



DATA MBIT FOR STANDARD 15 



." • t • ... . Data 

" ' Fonnula ' ' Source * 

(I) fof 26 closures with competitive goal AND competitive outcome 



# of 26 closures . • 



(11) f of 26 closures with competitive goal EOT noncompetitive outcome ; 



# of 26 closures 



(111) J of 26 closures with noncompetitive goal AND noncompetitive outcome 



fof 26 closures 



T 



(!v) fof 26 closures with noncompetitive goal BUT competitive outcome / 



i of 26 closures 



%■ ' 



-300 



/ c(i 

COMPUTING THE OflTft ELEMENTS 



\ 

r 



Standard Data Source Computation - ;, ' ; 

Standard 9: Modified Case Percentage of all,valf(l cases, who 

R-300 Validity Review Schedule achieved each data element ; 



Standard 10: Modified Case'. Percentage of all valldxases who 

Eligibility Review Schedule achieved |acb data element 



Modified Case \ Percentage of all valid to who 
Timeliness ' ^* Review Schedule - afchleved each data element 



Standard 12: Modified Case Percentage. of all valid cases 



IKRP Review Schedule achieve each data element 



Standard 13: 
Goal PIqrirring 



- 300 



Ratio for each of the four data 
elements 
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INTRODUCTION TO THE MODIFIED CASE REVIEW SCHEDULE 

• Does everyone have a packet that looks like 

this? ■■■ » - - . . ; 

• All the information to be discussed today is 
contained in this packet., pse it. as a reference 
guide both during and after the presentation. 
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. OBJECTIVES OF THE PRESENTATION ■ * 

Today's presentation has four objectives: 

To explain the purpose of the Modified Case 
Review Schedule; 

• To familiarize VR staff With the data elements 
contained in the MCRS and their relationship 
to t^e Procedural Standards; 

• To overview the sampling procedures . recommended 
for Selecting cases for. review; and, 1 \ | 

• To look at tfye qualifications and experience 
necessary for potential c£se reviewers. 



OVERVIEW OF THE MODIFIED CASE REVIEW SCHEDULE' 

TS - The Case Review Schedule is an instrument 

designed by the San Diego State University , 
to determine whether state VR programs are 
providing services in accordance with the 
regulations and guidelines mandated by- the? 
Rehabilitation Act of 1973. Based on this 
CRS, a Modified Case .Review Schedule V(MGRS) 
was developed by BPA to assess trie four Pro- 
cedural Standards relating to compliance with 
eligibility and the 'iWRP. We win now briefly 
review each of the sections Of the Modified 

.->'' Case Review Schedule and indicate their rela- 
tionship to the fouJ- relevant Procedural- Stan- 
dards. 



Section I .A and I .B : Identifying Information and 

Significant Case Data 

Section I. A provides information for: 

• identifying the client; and 

• merging the datawith other documents: 
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■**-,#• Section I:B records significant dates relevant to 
,• the client's program experience, such as date of 
^ . 'Service initiation, etc.'; 

*V These sections are used as Supplemental information 
' \ for Standards- 9 through 12. 

4 Sectibn \I.C : -R-300 Verification Instrument 

l . • This is used to assess the degree to which infor- 
mation submitted to RSA on critical items of the 
R-300 was corroborated by casefile information* 
Since the purpose of Standard 9 is to verify the 
^ 1 R-300 information, this will provide a manual con- 
firmation procedure. . 



5 s — ... Section II. A : Evaluation of ^Rehabilitation Potential: 
N < Preliminary 'Diagnostic Study - Status 02 

■::■(• /V ; • This section, used for Standard 10, assesses; the 

extent to which )btie case record documents' the-,- occur - 

. / rence of the. various Activities needed to conduct an 

v'/ effective preliminary diagnostic study. This 

v ; V .■ : . shoyld ^contain all of the information necessary' to 

■'„;; ,■ ■■ ,make an assessment of a client f s eligibility for VR 

-•' v ■ * services, such as : ; ■' . 

• [. • -v" , .. — medical reports; : 

— psychiatric examinations, etc. 

t * ■ * 

6 Section II. B : Evaluation of Rehabilitation Potential: 

Extended Evaluation - Status 06 

1 ...V In regards to Standard 10, this section seeks docu- 

mentation that the state agency has followed proper .. . 
procedures in placing applicants into; extended eval- 
uation (Status 06). In particular i the concerns are 
' ; that case records include: 

** 1) a certification for extended evaluation to 
determine rehabilitation potential; , 

'•■**/"'.• 2) the rationale for determining the client's need for 
• , ;s \ " extended evaluation; 

' ' 3) evidence of the occurrence of thorough assess"-' 
> ments of progress at least every 90 days; and 

** ,4) documentation of the eligibility decision 
resulting from extended evaluation. 

§ELC ./ 235 . 
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Provision of this information will help to ensure, 
that- extended evaluation isrused. ohiy when, appro- 
priate, that, clients' proceed in a timely manner., 
andL-£h#4i minimal recording/ needs for Status 06 
ar&;5SfSSntained. 

In regards to Standard 12, this section seeks to 
document that the IWRP's contain all of tfhe infor- 
mation required under Status 06. This information 
includies : / * - • 1 - T 

, ■ . "; ■ ' • 4' - ' .-,« 

1) a definition of the terms and conditions* £csr; / 
provision of services; , • • ■ "Vv-v y ;'• ' 

2) a documentation that the client was "informed " 
of his or her specific rights, including the 

VJ right to participate in the development of 
- ^ the program; . 

3) .an outline for the client's vocational goal 

arid : . a. .timeframe for its achievement ; ' N^'' * • ' : '- : r* 

.4) a specific set of evaluation procedures and 
criteria; 

5) a documentation of the final eligibility deci- 
sion and, for throse clients closed as ineligible: 

documentation^ that the client participated 
in the decision', and # 

. — documentation that provision was made for 
periodic review. 

Provision of this information helps ensure both the 
a&ierance^ to the IWRP provisions as well as the 
adherance to a timely movement of the client through 
Status 06. The information also ensures that clients 
are aware of their rights to continued services or 
review, if declared ineligible. 

Section III : Eligibility - Status 10 

The purpose of this section, which is relevant to 1 
Standard 10, is to document that a certification of 
eligibility was completed for each accepted client, 
and that counselor documentation in the case records 
confirms : - 

1) the existence of a disability; 

2) thfc existence of a substantial handicap to employ- 
ment ; and 

3) the likelihood that VR services will benefit the . 
client. 
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Section IV : Evaluation of Rehabilitation Potential: 

Thorough Diagnostic Study - Status 02 5 10 

• This section is used solely for problem identification. 
It includes questions on' the quality and scope of thie 
diagnostic study. 



10 T /• Section V : Individuali:ze4 Written Rehabilitation- 

• " ■ ."; V- ■• . \ Program - States 12 

. V .';>, ; :.'/•''. V This ;•; section is used for Stanlard. l2. Since the IWRP 
' is used to establish a counseior/clis^jt^alliance 
regarding the provision of services toward a specific 
vocational goal'/ it is important' tRat* the IWRP contains 
all of the information necessary for' establishing such 
an alliance. ' Section V documents:. 



** 1) that the client was informed of the terms and 
conditions for the provision of > services ; ' *" 

2) that the client was informed of client rights; 



5) that 'the client participated in the full plan- 
ning and review process;- and 

4) that the IWRP contains essential information/ 
r such as goals, time frames, e^luation procedures, 

, schedules, etc.* 

J* 

• Inclusion of this information in the IWRP clarifies, 
the roles, relationships, and duties of agency. and- ' . 
client ^toward achieving thie vocational A goal . 



11 Section VI :. Delivery of Services r 

Statuses 14; 16, 18, 20, 22, and 32 

• This section, used for Standard 12, when taken together 
>ith the information provided in Section V, describes 
the Overall VR profcess, consisting of: 



— the plan (i.e., the terijis, conditions, and infor- 
mation set f6rth in the IWRP needed to provide 
services ) ; and 

— the specific program of services undertaken to 
achieve the vocational goal embodied in the IWRP. 

If we know the extent to which planned services are 
actually delivered, we can then determine the extent 
of effective M follow through 11 on the service planning 
process. ■ • - 
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^2 * **■ Section Vli : Termination of Cases, 



• This section, in relation to Standard 10, focuses on 
nonsuccessful closures. For these closures, Standard 
10 attempts to assess the following compliance issues: 

..*v * 
1) Does . the case record document? the ineligibility/ 
tenninatibn decision, aid? the 4 basis for that 
decision?' ■ 



** . 2) Have clients been granted their legal rights to 
participate in the ineligibili^y/terrainatdon 
decision? v - ; K 

** 3) Have clients been informed of their right to an - 
annual review of the decision? and ? _ 

** 4) Have the'xfequired annual reviews occurred, and 
the results been documented? ■ 

The need for this information is twofold: 

:./:V l ■ ; .1) Assuming that supervisory ^personnel will want/ to 

review cases invplvinff^ineligibility or unsuccess- 
ful termination, they\hould have the ability to 
review cases sampled at random or target reviews 
to particular counselors. 

2) It assures clients' protection by providing evi- 
dence that the necessary steps have occurred with 
the knowledge and participatioh of the'client. 

j 

[ • Informed of their rights to review, hopefully these 

clients will re-enter the system later and be success- 
fully rehabilitated. 

13 In relation to Standard 12, this 'section seeks to ensure: 

— that the rationale for closure decisions are 
recorded on the IWRP; and 

that the client (or his/her appropriate represent- 
ative) was consulted prior to the closure decision. 
•„.■ . * ' ' - 

• Finally, several"' questions^jfertaining to 26 closures 
are included in Section VII. These questions; are 
related to problem identification. v 
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Section VIII : Timeliness Assessment Instrument 

This section responds directly to Standard 11, It 
is designed to lipk subjective assessments of the 
timeliness of case movement to 'objective data on the 
length of tim,e spent in various statuses by different 
disability types. The assessments are used in con- 
junction with data on client characteristic^ and 
services provided to investigate how agencies might 
-avoid undue delays in the service process. 

TS - Having reviewed all sections. in the Modified 
Case Review Schedule, let us turn to the ques- 
tion of sample selection and frequency of 
review. 

SAMPLING PROCEDURE 

The sampling procedure is related to two practical > 
concerns: 

• that a sufficient number of cases be reviewed 
to allow for analysis and interpretation; and 

• that reviews be conducted on clients who are in 
service, thereby allowing for the assessment of 
current operations and the application of cor- 
rective measures should problems exist. « 

There are three issues of concern here: 

1) It* is important that the variable "entered/did . ] 
not enter extended evaluation 11 be included among 
the sample selection. This will allow assess- 
ment of compliance with regulations pertaining 

to ineligibility determinations. 

2) It is important that the sample selection 
criteria ensure that some cases which fall . 
into all £f the * following categories of ser- 
vice process will be represented in the sample: 

V eligibility 

•> extended evaluation 

• -plan development 

• service provision / ^ 

• closure. 
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This Will allow for the full exploration 'of all 
issues represented by , the four Procedural Standards 
. and will guarantee that reviews will be conducted 

von in-service cases. ._/-'<, 

- . ** 3) The final issue, concerns the physical location 
v. 7 of the data collection effort. Removal of case 
v files from the district offices may. prove prob- 
iV' lematic, particularly in cases where the files 

. are currently in use. A system whereby t reviewers 
* v travel to the various districts to conduct reviews 

may be preferable. 

16 FREQUENCY OF REVIEW " \ 

Statewide assessments should occur every three years, 
and never less than four. If possible, they should 
be supplemented by more frequent , targeted spot checks 

*TS - Now let us review the qualifications and exper- 
ience necessary for the case reviewers. 

" " '■ 17 * ■ CASE REVIEWER QUALIFICATIONS 

• Since the quality and resulting data is strongly- • 
influenced by the -capabilities of the people who 
conduct the reviews, there are four main qualifica- 
^ tions an effective reviewer should possess : 

,1) An intimate familiarity with a wide variety 
' of aspects of operations in that sfate agency. 

2\ Experience in casework, such as counseling, 
casework supervision, and overall operations 
and administration. 

3) Objectivity when reviewing cases, and 

- ji* ' ■.. i ■*'•■*♦ . 

4) The ability to devote 100% of their time to 
these responsibilities. This will ensure 
that, their., familiarity with the task will be 
reinforced, their reviewing ability will be 
improved, and that^the necessity for intensive 
training with each new data collection cycle 

. wj 11 be obviated. 
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CONCLUDING COMMENTS 



This concludes our formal presentation on the MCRS. 
As we mentioned .at the beginning, all of the infor- r 
mation presented today is included in the Trainee 
JJandbook, Section E. 
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OBJECTIVES OF THE PRESENTATION . - 

• Explain the purpose of the MCRS;, < 

• Discuss data elements- and their relationship 
to Procedural Standards; 

• Overview sampling procedures; 

. & ' ; 9 * ; •- . 

• Look at qua 1 i f i cat i ons of rev i.ewers . 
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- SECTIONS I, A, and Identifying Information and Significant Cose Data 



Section LA-.: 



Identification of the client 



merging the data with other sources 



Section LB,: 



significant dates' 



• , l)sej| for: 

• 'Standard 9 - 5-30*0 Validity 

• Standard 10 - Eligibility 

■ ', > ■ i t * ' ' 

8 Stdndard 11 - Timeliness 
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SECTION I.C : R-500 Verification Instrument 



Assesses' correspondence. between R-300 and case file data. 



Used for: 

Standard 9 - R-300 Validity 



* . 
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SECTION 1 1. A, : Evaluation of Rehabilitation Potential : 
v Preliminary Diagnostic Study - Status 02 

0 t ? — ; 



• Medical Report 

• Psychiatric Examination 

9 * 



Used for: 



Standard 10 - Eligibility 



'■■i 



B_,: Evaluation of Rehabilitation Potential : 
Extended Evaluation - Status 06 

/ . 

x '' 

■ 

• certification for extended evaluation; 

• basis of the need; 

• evidence of 90-day assessments; and 

• documentation of eligibility decision, 

Used for: . 

Standard 10 - Eligibility-* 




SECTION II.B. ( cont i nued ) 



• definition of terms and conditions for services; 

• documentation, that client was informed of rights; 

• outline of .vocational goal and time frame; 

• evaluation procedures and criteria; and 



documentation of eligibility decision. 



Used- for.- 



Standard 12 - IWRP 
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Mi: Eligibility Status 10 

* : : . "' 
. <*' 

• existence pf disability^- 

• existence of substantial handicap; and 

• likelihood of benefit from VR, 

Used for: 
Standard 10 - Eligibility 

* 

• .1 



25u 



7 > \ 

SECTION IV ; Evaluation of/ Rehabilitation Potential : 

Thorough Diagnostic Study - Status 02 and 10 



Quality and scope of diagnostic study 



Used for: 



Problem Identification 
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SECTION V: IWRP - Status 12 



client Was Informed of terms and conditions for service; 



j» client waslTiformed of rights; 



client participated in plan; 



IWRP contains essential Information such as: 



-- goals 



-- time frames 



s J 



evaluation procedures, etc, 

■ • 1 . 




- IWRP 



< * 



used with Section V to describe, overall process; 

'"3 

.'I 

includes the plan and specific program of services; and 

■ 

measures "follow-through." 



: Termination of Cases 



documentation of decision; 



• client partrcipation In decision 

I . " < - , ■ 

• right to annual review; -and* 



• documentatton' of r 



Used for: 




, Standard 10 - Eligibility. " v 

/ . . . , . . . 



SECTION VI 1 (continued) ' f "" 9 



,E (£ 



• documentation of rationale for closure; 

• client consulted prior to decision 



Used for: 



Standgrd 12 - IWRP 
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SECTION VIII; Timeliness Assessment Instrument 



» links subjective assessments of case movement to 
objective data on time in status; and 



? used in conjunction with client and service data 
to develop ways to avoid delays, 



Used for: 



Standard 11 - Timeline 



SAMPLE SELECTION ' . 

• Sample Selection should Include:,' 

1) Both "entered" and "did not enter 
extended evaluation; 

ft 

2) A cross-section of all statuses; 

3) East access to case files, 



FREQUENCY OF REVIEW 
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Ideally, every three .&ar$^ 



Never less frequently than every four years 




Supplement with spot checks 



ft 
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QUAL I F I CAT I ONS ; AND EXPERIENCE OF CASE REVIEWERS 



if 

■' l . V' 



■■V ."* •• 



nfearUy with all aspects of state agency operations 



,;4^^r|ef?Ge in casework counseling, casework 

tn, ope rat ions , dh#adm iqlst r& Ugn ) , 





V Exclusive Fespoft'sfbiHty .for the review 1 process. 
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